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TELEMAMMOGRAPHY  SYSTEM:  A  MODEL  FOR  A 
REGIONAL  SYSTEM 

ELLEN  SHAW  de  PAREDES,  M.D. 

SAMUEL  J.  DWYER,  Ph.D. 

MEDICAL  COLLEGE  OF  VIRGINIA,  RICHMOND,  VA 
UNIVERSITY  OF  VIRGINIA,  CHARLOTTESVILLE,  VA 


PURPOSE: 

The  research  hypothesis  being  tested  is  that  a  telemammography  system  provides  a 
mechanism  for  digitizing,  transmitting,  archiving,  and  displaying  mammograms  so  that  a  trained 
radiologist  utilizing  grayscale  monitors  (2k  x  2k  x  8/12  bits)  can  interpret  the  images  with  an 
accuracy  level  sufficient  for  primary  diagnosis.  A  new  metric  for  measuring  the  performance  of 
telemammography  systems  has  been  developed,  throughput/cost  ratio.  This  metric  provides  a 
measure  for  comparing  analog  to  digital  mammography  systems.  This  metric  has  been  employed 
in  comparing  manufacturing  processes,  i.e.  jobs/sec/cost.  A  ROC  curve  analysis  (1-6)  is  utilized 
to  compare  the  accuracy  of  interpretation  of  analog  images  versus  digitized  images. 

The  hypothesis  is  being  tested  by  utilizing  a  laser  film  digitizer  with  a  50-micron  pixel  size 
and  by  performing  ROC  studies  to  compare  conventional  analog  screen-film  mammography  with 
digitized  screen-film  mammograms  displayed  on  grayscale  workstations  (two  monitors,  each  2k 
X  2k  X  8/12  bits).  The  wide  area  network  (WAN)  being  utilized  are  terrestrial  and  satellites  links. 
The  goal  of  this  study  is  to  determine  the  requirements  to  deliver  high  quality,  high  resolution 
mammography  images  from  remote  locations  that  may  not  have  a  terrestrial  data 
communications  infrastructure  available. 

DESCRIPTION  OF  EXPERIMENTAL  PROCEDURES: 

A  series  of  retrospectively  collected  200  normal  mammograms  and  200  abnormal 
mammograms  containing  a  single  lesion  including  masses,  calcifications,  and  asymmetric 
densities  (focal  area  of  asyirimetry  or  architectural  distortion)  were  collected  for  conducting  a 
ROC  analysis.  The  mammographic  and  pathologic  findings  were  classified  using  the 
terminology  of  the  American  College  of  Radiology  Lexicon.  The  ROC  study,  which  used  12 
readers  compares  analog  screen-film  mammograms  and  the  mammograms  which  were  digitized 
at  50  microns  spot  size.  The  digitized  analog  screen-film  mammograms  were  archived  into  a 
database.  A  comparison  ROC  curve  analysis  of  the  digitized  mammograms  is  being  conducted 
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by  displaying  them  on  a  2k  x  2k  x  8/12  bit  grayscale  workstation  using  two  monitors.  An 
evaluation  performance  is  being  conducted  using  the  metrics  of  throughput  and  throughput-to- 
cost-ratio.  The  throughput  and  throughput/cost  ratio  is  based  on  the  method  of  Jackson  Network 
Analysis  and  has  been  completed  on  the  analog  system.  The  calculations  are  based  on  a  resource 
allocation  table  in  which  the  average  time  required  for  each  step  is  measured  as  applicable  for 
each  resource.  A  satellite  transmission  of  mammograms  is  underway  between  the  University  of 
Virginia,  the  Cleveland  Clinic  Foundation,  and  the  NASA  Lewis  Research  Center  (ACTS 
Project). 

A  softcopy  display  protocol  has  been  developed  and  is  being  implemented  to  review  the 
images  in  a  similar  format  to  the  manner  in  which  they  are  interpreted  on  a  viewbox. 

SUMMARY  OF  RESULTS  TO  DATE: 

An  ROC  analysis  has  been  conducted  on  the  400  analog  screen  film  mammograms.  For  all 
readers  the  results  of  the  ROC  analog  images  varied  10%  from  the  0.83  area  under  the  ROC 
curve,  which  is  a  desirable  result,  reflecting  proper  case  selection.  An  ROC  analysis  is  underway 
for  the  two  monitor,  grayscale  2k  x  2k  workstation  display  of  the  digitized  mammograms.  The 
results  of  the  throughput  analysis  of  the  analog  images  showed  that  the  technologist  was  the 
bottleneck  (0.041  jobs/minute)  and  the  throughput  to  cost  ratio  was  0.001203.  Our  measure  of 
luminance  of  the  workstation  shows  the  average  luminance  level  to  be  60-80  foot-lamberts 
compared  to  400  to  600  foot-lamberts  (with  film)  that  of  the  mammography  viewbox.  The 
contrast  range  for  the  grayscale  display  is  18-20%  lower  than  that  of  analog  images. 

The  use  of  the  soft-copy  display  stations  has  thus  far  been  discouraging.  While  the  ROC 
analysis  results  are  not  that  different,  the  users  are  vocal  in  their  disappointment  in  the  display 
workstation  graphics.  Viewing  times  are  in  the  range  of  8  to  10  minutes  for  screening  (with  four 
views).  The  digitized  images,  with  proper  window/level  settings  can  be  made  diagnostic,  but  the 
trade-ofif  is  that  it  takes  time.  This  would  suggest  that  a  perceptually  linearized  display  should  be 
presented  to  the  user. 

On  the  basis  of  our  studies  to  date,  to  be  clinically  useful,  telemammography  must  provide 
user-friendly,  rapid-throughput  soft-copy  interpretation  of  digitized  screen-film  mammograms. 
Telemammography  could  enable  second-opinion  interpretation  by  providing  that  mammography 
expertise  available  from  community  practices. 

CONCLUSIONS: 

The  resulting  soft-copy  display  protocol  for  the  digitized  screen  film  mammograms 
reduces  the  reading  throughput  by  a  factor  of  one-half.  The  digitized  screen  film  images  (4k  x  4k 
X  12  bits)  presents  a  difficult  display  protocol  for  a  2k  x  2k  x  8/12  bits  monitor.  Additional 
challenges  to  display  are  the  presentation  of  4  images  on  2  monitors  as  well  as  the  review  of  prior 
images  for  comparison.  Because  of  the  lower  contrast  range  of  the  grayscale  displays,  image 
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processing  (window  and  level  functions)  is  needed  on  the  softcopy. 

METHODS: 

Image  Selection. 

A  database  of  200  normal  screen-film  mammography  examinations 
and  200  abnormal  screen-film  mammography  examinations  have  been  collected. 

Each  examination  contains  the  following:  image  study  performed;  results;  description  of 
mammographic  findings  if  abnormal  (using  the  BIRADS);  pathology  report  of  biopsy  performed; 
demographics;  and  radiation  exposure.  The  abnormal  films  comprise  a  representative  sample  of 
pathological  diagnosed  breast  lesions  in  the  population  served  by  the  University  of  Virginia  and 
Medical  College  of  Virginia.  Lesions  were  classified  according  to  the  BIRADS  lesion  for  breast 
density  and  finding  description  by  an  expert  panel.  The  normal  films  have  been  substantiated  by 
normal  elinical  follow-up  of  at  least  2  years’  duration  and  are  matched  to  the  overall 
parenchymal  density  of  the  abnormal  films.  All  patient  data  and  names  were  blocked  and  a  case 
number  was  assigned  to  each. 

Appendix  I  contains  a  listing  of  each  case,  by  case  number,  in  the  first  column.  The 
second  column  lists  each  case  by  the  pathology  report.  Column  3  lists  the  response  by  the  expert 
panel  to  the  question  of  diagnosis  of  the  image.  Colunm  4  is  the  expert  panel  response  to  the 
question  of  micro-calcifications;  column  5  is  the  expert  panel  response  to  the  question  of  mass 
present;  and  column  6  is  the  expert  panel  response  to  the  question  of  focal  area  of  asymmetry  or 
architectural  distortion. 

DIGITIZED  IMAGE  DATABASE 

The  eollected  database  of  analog  mammographic  films  was  converted  into  digital  arrays 
using  a  laser  film  digitizer  with  a  50  micron  pixel  spot  size  and  12  bits  per  pixel  (Lumisys, 
Sunnyvale,  CA,  Model  150  with  a  50-niicron  spot  size).  The  laser  film  digitizer  was  connected 
to  a  sun  workstation  (SPARC  Model  40)  and  connected  to  an  internet  connectivity.  All  patient 
data  was  removed  and  a  case  number  assigned  to  each  digitized  case.  This  image  database  is  on 
an  MOD  optical  disc  and  portions  are  recorded  on  a  CD-ROM  for  use  by  other  researchers.  The 
digitized  images  are  archived  into  a  database  and  archived  on  both  a  hard  drive  disk  and  a  Tape 
drive. 


The  laser  film  digitizer  uses  an  analog  film  SMPTE  pattern  once  a  week  to  provide  an 
adequate  quality  control.  Additionally,  the  laser  film  digitizer  is  calibrated  once  a  week  using  the 
protocol  provided  by  the  manufacturer.  Each  digitized  image  was  displayed  on  a  grayscale 
display  workstation  to  assure  proper  digitization  of  the  screen-film  images. 

We  have  performed  histogram  analysis  on  a  sampled  population  of  the  digitized  image 
database.  As  expected  of  laser  film  digitizers,  the  dynamic  range  of  the  gray  scale  was  narrow. 
To  insure  that  soft-copy  readings  would  provide  a  reasonable  dynamic  range,  we  displayed  each 
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digitized  mammogram  and  adjusted  the  interactive  window/level  settings  to  insure  a  proper  soft- 
copy  display.  To  insure  that  the  digitized  images  "looked  like"  the  original  screen-film 
mammograms,  we  set  the  window/level  settings  and  printed  out  the  digitized  images  on  both  the 
Koday  and  AGFA  laser  film  printers.  Appendix  II  provides  a  sample  of  this  work  illustrating 
both  the  original  and  the  corresponding  laser  digitized  screen-film.  The  border  of  the  digitized 
screen  film  mammogram  can  be  seen  on  the  laser  printed  digitized  image.  No  ROC  readings 
were  conducted  on  Aese  laser  printed  films.  Rather,  they  were  used  only  as  a  quality  control 
protocol. 

IMAGE  GRADING  AND  READING. 

The  readers  use  a  5-point  graded  response.  Each  reader  completed  a  worksheet  form  for 
each  image,  requiring  a  response  to  the  following  four  questions  by  circling  the  selected  graded 
response.  The  pathology  reports  of  the  biopsies  served  as  ground  truth. 

Masses: 

1 .  (definitely  not  present):  2.  (probably  not  present); 

3.  (equivocal;  4.  (probably  present);  5.  (definitely  present) 

Microcalcifications: 

1 .  (definitely  not  present);  2.  (probably  not  present); 

3.  (equivocal);  4.  (probably  present);  5.  (definitely  present). 

Focal  areas  of  asymmetry  or  architectural  distortion: 

1 .  (definitely  not  present);  2.  (probably  not  present); 

3.  (equivocal);  4.  (probably  present);  5.  (definitely  present). 

Diagnosis  of  image: 

1.  (definitely  benign);  2.  (probably  benign); 

3.  (equivocal);  4.  (probably  malignant);  5.  (definitely  malignant) 

Images  were  presented  to  the  set  of  readers  first  as  analog  mammographic  films.  The 
films  were  hxmg  in  sets  on  several  rollerscopes  in  both  departments.  The  readers  read  each  image 
and  responded  to  the  above  four  questions.  The  readers  may  utilize  a  magnifying  glass  as  an 
option  if  they  wish.  The  readers  are  now  reading  the  digitized  film  images  on  a  grayscale 
workstation.  It  is  believed  that  enough  time  has  passed  from  the  analog  screen  film  readings  that 
the  readers  will  not  remember  the  cases.  A  random  number  generator  is  being  used  to  modify  the 
order  of  presentation  on  the  grayscale  display  as  compared  to  the  presentation  of  the  analog 
mammography  images.  Reader  data  is  recorded  in  a  notebook,  one  page  per  mammographic 
exam.  The  reader  data  is  then  entered  into  a  database  and  a  ROC  analysis  performed  for  each  of 
the  four  questions  responded  to  by  the  readers. 

We  are  disappointed  with  the  interactive  user  operation  of  the  soft-copy  display 
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workstation.  The  chief  difficulty  is  the  7  to  1 0  minutes  for  reading  each  case.  In  part  this  long 
time  is  a  result  of  not  having  an  automatic  method  for  adjusting  each  window/level  case  setting. 
The  radiologist  is  required  to  load  the  case  into  the  display  buffer  (4  to  5  minutes  from  CD-ROM 
storage),  then  set  the  window/level  adjustment,  and  then  read-out  the  case.  This  technology 
bottleneck  has  seriously  limited  those  radiologists  required  to  read  the  soft-copy  display  in  order 
to  compare  their  results  to  the  original  screen-film  images.  In  addition,  we  often  suffer  display 
station  crashes,  necessitating  a  restart  for  the  soft-copy  display. 

STATISTICAL  CONSIDERATIONS 

In  selecting  the  sample  size,  we  have  used  two  methods  to  increase  the  reliability  of  the 
calculation.  In  the  first  method,  we  have  chosen  type  I  error  to  be  equal  to  0.05  (1.96)  for  a  two 
tailed  test  and  type  1 1  error  to  be  equal  to  0.20  (0.84).  Hence  the  power  index  (PI)  is  2.8.  The 
sample  size  is  given  by  n  equal  to  ((PI  x  SE/  (A1  -  A2))  squared,  where  A1  is  the  area  under  the 
ROC  curves  for  analog  films  and  A2  is  the  area  under  the  ROC  curves  for  digitized  and  grayscale 
displayed  images.  We  estimate  the  standard  error  to  be  0.5  for  12  readers,  all  of  whom  provide 
primary  coverage  in  mammography.  We  assume  the  differences  in  the  means  of  the  areas  under 
the  two  ROC  curves  to  be  10%.  Then  the  nximber  of  samples  per  population  is  196  or 
approximately  200  pairs  of  images.  Hence  we  require  200  normal  and  200  abnormal 
mammograms. 

In  a  second  method  that  confirms  the  findings  of  the  first  approach,  we  assume  the 
following;  alpha  error  0.05,  beta  error  0.2,  power  80%,  and  the  expercted  difference  between  the 
areas  underneath  the  ROC  curves  for  the  digital  (A2),  and  analog  (Al)  mammographic  films  as 
0. 1 .  This  expected  difference  is  the  minimum  difference  that  we  consider  clinically  important. 
Specifically,  we  are  estimating  Al  to  be  0.85  and  A2  to  be  0.75.  Since  each  reader  will  review  the 
same  sample  of  cases,  we  will  use  a  correlation  coefficient  conservatively  estimated  at  0.3.  Using 
the  methods  of  Hanley  and  McNeil  and  a  two-sided  test  of  significance,  we  calculate  that  with  a 
single  reader  we  will  require  120  cases  each  of  abnormal  and  normal  films.  Thus,  if  the 
difference  between  Al  and  A2  is  as  estimated  above,  we  will  have  a  greater  than  80%  chance 
with  a  single  reader  of  detecting  a  statistically  significant  difference  in  diagnostic  performance 
with  120  pairs  of  normal  and  abnormal  cases.  Using  additional  readers  (12  total)  and  cases  (200 
each  of  normal  and  abnormal)  will  give  us  a  greater  likelihood  of  detecting  a  significant 
difference. 

A  ROC  curve  analysis  of  the  reader  data  is  accomplished  by  using  applications  software 
provided  by  Charles  Metz  of  the  University  of  Chicago/University  of  Piftsburgh.  The  software  is 
operational  on  a  SUN  workstation  at  UVA.  Two  ROC  curves  for  each  reader  for  each  of  the  four 
reader  questions  will  be  generated,  one  for  the  analog  mammographic  films  and  one  for  the 
digitized  mammograms  displayed  on  the  grayscale  display  workstations  on  our  AGFA  image 
display  workstation.  The  index  of  performance  will  be  the  area  under  the  ROC  curve  for  each 
reader.  In  the  event  of  a  degenerate  ROC  curve,  we  analyzed  the  data  using  two  approaches. 

One  approach  utilized  the  Wilcoxon  statistic  to  estimate  the  area  underneath  the  ROC  curve  and 
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the  standard  error.  The  second  approach  involved  a  third-order  spline  curve-fitting  software 
program  implemented  at  the  University  of  Virginia. 

The  ROC  analysis,  thus  far  completed,  is  shown  in  Appendix  III,  ROC  Analysis.  A  table 
is  shown  as  to  the  Reader  List.  Readers  listed  as  3,  5, 7, 12  have  left  the  study.  These  readers 
have  been  contacted  but  we  only  have  one  workstation,  now  located  at  the  University  of 
Virginia.  We  have  been  approved,  at  not  funding  increase,  to  extend  the  study  until  March, 

1999.  We  hope  to  obtain  more  readings  from  the  display  station  in  spite  of  its  serious 
limitations. 

MAMMOGRAPHY  GRAYSCALE  READING  WORKSTATION 

The  grayscale  workstation  for  the  project  is  a  AGFA  review  station  (IMP AX,  RS3000, 
Ridgefield  NJ)  which  is  comprised  of  a  SUN  Ultra  2  with  192  Mbytes  of  RAM,  4  Gbytes  of 
internal  HD  storage,  two  DOME  high-resolution  video  cards,  and  two  high-resolution  (2k  x  2.5k) 
Orwin  grayscale  monitors.  The  system  is  running  Solaris  2.5,  with  Common  Desktop 
Environment  installed  and  uses  the  Motif  Graphical  User  Interface.  The  Review  station  software 
is  DICOM  3.0  compliant. 

We  added  a  Radion  4Gbyte  external  hard  disk  drive  for  storing  images  from  the  current 
reading  set,  and  a  Pinnacle  Micro  Vertex  2.3  Gbyte  Magneto-Optical  drive  as  a  transfer/backup 
device. 

A  separate  instance  of  the  SMV  software  was  started  on  the  desktop  of  each  monitor,  and 
expanded  to  full  screen.  This  allowed  two  images  to  be  displayed  at  once.  Initially,  the  LMLO 
image  was  displayed  on  the  left  monitor,  and  the  RMLO  was  displayed  on  the  right  monitor. 

This  was  followed  by  the  LCC  and  the  RCC,  respectively,  and  then  images  were  displayed  as 
requested  by  the  radiologist  reader. 

DISPLAY  WORKSTATION  SOFTWARE 

The  software  being  used  is  SMV,  a  program  wriften  by  Prof  Marty  Stanton  of  Brandeis 
University.  Originally  designed  for  crystallography  work.  Prof  Stanton  adapted  it  to  the  needs  of 
the  Telemammography  project  by  giving  it  the  ability  to  directly  read  the  images  created  by  the 
Lumisys  150  laser  film  digitizer.  This  made  it  possible  to  display  the  images  without  going 
through  the  intermediate  step  of  converting  our  data  to  the  DICOM  format  needed  by  the 
IMP  AX  database  of  the  AGFA  review  station. 

SMV  supports  all  of  the  image  processing  techniques  needed  for  the  display  of  the 
mammography  images,  including  panning,  zooming,  window-leveling,  and  grayscale  inverting. 

It  also  provides  features  that  were  used  extensively  to  prepare  the  images  for  display,  particularly 
rotation  and  cropping. 
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Two  factors  had  to  be  considered  when  preparing  the  images  for  display .  One  was  that 
patient  information  recorded  on  the  mammography  film,  as  well  as  indication  of  the  source 
hospital  for  the  films,  had  to  be  masked.  The  other  was  that  a  reduction  in  the  overall  size  of  the 
images  was  desirable,  to  decrease  storage  requirements  and  reduce  the  amount  of  time  required 
to  load  and  display  the  image.  To  address  both  of  these  concerns,  SMV's  cropping  feature  was 
used  to  reduce  the  image  to  just  the  part  of  the  film  that  actually  held  the  image  of  the  breast. 

This  greatly  improved  performance  of  the  system,  as  well  as  satisfying  the  protocol  requirement 
that  no  patient  or  hospital  information  be  visible  to  the  reader. 

DISPLAY  PROTOCOLS  FOR  WORKSTATION 

The  Display  Protocol  for  the  AGFA  workstation  consists  of  four  displays:  Display  A  (LMLO 
on  left  screen  and  RMLO  on  right  screen);  Display  B  (LCC  on  left  screen  and  RCC  on  right  screen); 
Display  C  (LMLO  on  left  screen  and  LCC  on  right  screen);  and  Display  D  (RMLO  on  left  screen 
and  RCC  on  right  screen).  Display  A  and  B  were  presented  to  the  reader  and  if  requested  then 
Display  C  and  Display  D  were  also  presented. 

FUTURE  EFFORTS 

During  the  extended,  non-additional  funding  period,  we  will  accomplish  the  following: 

1 .  Complete  reading  the  majority  of  the  400  cases  on  the  AGFA  soft-copy  display 
workstation. 

2.  Complete  the  ROC  analysis 

3.  Complete  the  NASA  ACTS  telemammography  readings. 

4.  Complete  the  throughput  workflow  and  modeling  analysis. 

NASA-LERC  ACTS  TELEMAMMOGRAPHY  PROJECT 

The  ACTS  telemammography  project  a  joint  effort  between  NASA's  Lewis  Research 
Center,  the  Cleveland  Clinic,  and  the  University  of  Virginia  Department  of  Radiology.  ACTS  is 
NASA’s  Advanced  Communications  Technology  Satellite.  Earth  stations  have  been  installed  at 
the  three  locations.  Unix  workstations  are  connected  to  the  earth  stations  at  each  location.  The 
purpose  of  the  project  is  to  test  the  efficacy  of  using  a  satellite  link  as  a  means  of  transmitting 
mammography  images  to  distant  locations.  The  experimenters  are  transmitting  digitized 
mammography  images  via  ACTS  satellite  between  UVa,  NASA-LERC  and  the  Cleveland  Clinic 
with  various  techniques  and  levels  of  compression.  Radiologists  at  Cleveland  Clinic  are 
examining  the  images  to  determine  their  quality  and  diagnostic  effectiveness.  The  current  phase 
of  tiiis  project  is  expected  to  run  through  the  spring  of  1999. 
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THROUGHPUT  ANALYSIS 


The  performance  metrics  being  used  in  our  Telemammography  modeling  are  diagnostic 
accuracy  and  throughput.  Diagnostic  accuracy  is  obtained  by  a  ROC  analysis.  The  area  under 
the  ROC  curve  is  the  probability  of  correctly  distinguishing  -  on  the  basis  of  the  diagnostic  test 
results  alone  -  a  randomly  selected  disease  case  from  a  randomly  selected  non-diseased  case. 
Throughput  is  defined  as  the  rate  jobs-per-minute)  at  which  requests  can  be  serviced  by  the 
system.  The  throughput  of  a  system  generally  increases  as  the  load  on  the  system  initially 
increases,  without  the  addition  of  more  resources;  after  a  certain  load,  the  throughput  stops 
increasing  due  to  one  or  more  bottlenecks.  Throughput  analysis  is  accomplished  by  listing  the 
resources  used  and  the  steps  required  to  complete  one  job  together  with  the  average  time  per  step. 
The  resource  with  the  smallest  jobs-per-minute  over  the  complete  job,  is  termed  the  bottleneck 
resource. 


Table  1  and  Table  2  are  the  resource  utilization  tables  used  for  conducting  throughput  analysis, 
generated  for  MCV  (Medical  College  of  Virginia)  and  Stony  Point  (an  MCV  outpatient  clinic), 
both  in  the  Breast  Imaging  Section  of  the  Department.  The  resources  being  modeled  in  Table  1 
and  Table  2  are:  The  Clerk,  the  Technologist,  the  Imaging  modality,  the  Film  Processor,  the  File 
Room,  the  Resident,  and  the  Radiologist.  At  each  step,  the  average  time  to  complete  the  step  is 
measured  (minutes).  The  Throughput  for  each  resource  is  calculated  and  recorded  at  the  bottom 
of  each  table.  For  example,  in  Table  1 ,  the  throughput  of  the  Tech  is  calculated  by  1/(13 .2  +  4.7 
-1- 1.6  +  5.13)  =  0.041  jobs-per-minute.  This  calculation  assumes  that  the  Tech  is  working  at 
100%  capacity  and  can  work  no  faster.  In  Table  1,  the  system  bottleneck  is  the  Tech  since  0.041 
jobs-per-minute  is  the  smallest  throughput.  Likewise,  in  Table  2,  the  Tech  is  again  the 
bottleneck  at  0.071  jobs-per-minute. 

One  of  the  tasks  to  be  accomplished  is  the  use  of  modeling  techniques  for  a  regional 
telemammography  system.  This  section  of  our  report  describes  the  modeling  schemes  we  have 
developed. 
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IHUIC  li 

STEP 

CLERK 

TECH 

MODALITY 

FILM  PROCESSOR 

FILM  ROOM 

RESIDENT 

RADIOLOGIST 

REGISTRATION 

11.5 

0 

0 

0 

0 

0 

0 

PRIOR  FILM  AND/OR  FILM 
RETRIEVAL 

0 

0 

0 

0 

1.75 

0 

0 

IMAGE  ACQUISITION 

0 

13.2 

13.2 

0 

0 

0 

0 

FILM  PROCESSED 

0 

4.7 

0 

4.7 

0 

0 

0 

QUALITY  ASSURANCE 

0 

1.6 

0 

0 

0 

0 

0 

PRE-ACQUISITION  AND 
PROCESS 

0 

5.13 

5.13 

5.13 

0 

0 

0 

FILM  HUNG 

0 

0 

0 

0 

0.39 

0 

0 

REVIEW  OF  CLINICIAN 
INFORMATION 

0 

0 

0 

0 

0 

0.25 

0 

FILMS  READ 

0 

0 

0 

0 

0 

0.62 

0.62 

ADDITIONAL  VIEWS, 

STUDIES 

WRITE  EARLY  READING 

0 

0 

0 

0 

0 

0.34 

0 

REPORT  DICTATION 

0 

0 

0 

0 

0 

1.22 

0 

NOTIFY  CLINICIAN 

0 

0 

0 

0 

0 

1.43 

0 

REVIEW  RESULTS  WITH 
PATIENT 

0 

0 

0 

0 

0 

0 

1.34 

COMPARISON  WITH  PRIOR 
FILM 

0 

0 

0 

0 

0 

0 

0.44 

DICTATE  ADDENDUM  TO 
REPORT 

0 

0 

0 

0 

0 

0 

0.81 

FILING  OF  REPORT 

0 

0 

0 

0 

0.46 

0 

0 

THROUGHPUT 

0.067 

0.041 

0.055 

0.102 

0.384 

0.259 

0.311 

Bottleneck  is  the  Technogist  (0.41  jobs  per  minute). 
Table  2:  Stony  Point 


STEP 

CLERK 

TECH 

MODALITY 

FILM  PROCESSOR 

FILM  ROOM 

RESIDENT 

RADIOLOGIST 

REGISTRATION 

13.6 

0 

0 

0 

0 

0 

0 

PRIOR  FILM  AND/OR  FILM 
RETRIEVAL 

0 

0 

0 

0 

0.38 

0 

0 

IMAGE  ACQUISITION 

0 

7.3 

7.3 

0 

0 

0 

0 

FILM  PROCESSED 

0 

4.7 

0 

5 

0 

0 

0 

QUALITY  ASSURANCE 

0 

0.48 

0 

0 

0 

0 

0 

PRE-ACQUISITION  AND 
PROCESS 

0 

1.3 

1-3 

1.3 

0 

0 

0 

FILM  HUNG 

0 

0 

0 

0 

0.52 

0 

0 

REVIEW  OF  CLINICIAN 
INFORMATION 

0 

0 

0 

0 

0 

0.3 

0 

FILMS  READ 

0 

0 

0 

0 

0.96 

0.96 

ADDITIONAL  VIEWS, 

STUDIES 

WRITE  EARLY  READING 

0 

0 

0 

0 

0 

0.35 

0 

REPORT  DICTATION 

0 

0 

0 

0 

0 

1.09 

0 

NOTIFY  CLINICIAN 

0 

0 

0 

0 

0 

0.14 

0 

REVIEW  RESULTS  WITH 
PATIENT 

0 

0 

0 

0 

0 

0 

0.84 

COMPARISON  WITH  PRIOR 
FILM 

0 

0 

0 

0 

0 

0 

0.63 

DICTATE  ADDENDUM  TO 
REPORT 

0 

0 

0 

0 

0 

0 

0 

FILING  OF  REPORT 

0 

0 

0 

0 

0.43 

0 

0 

THROUGHPUT 

0.074 

0.07 

0.116 

0.159 

0.757 

0.353 

0.412 

Bottleneck  is  the  technologist  (0.071  jobs  per  minute 
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MODELING 


A  good  model  of  an  examination  will  predict  the  usefulness  of  the  exam.  The  parameters 
of  the  model  will  have  to  be  estimated.  A  good  model  will  reduce  the  data  required  to  determine 
diagnostic  accuracy,  throughput,  and  cost. 

1.  ROC  Analysis 

Analysis  is  based  upon  a  two-by-two  contingency  table  (Figure  1).  The  symbols  are:  ab  is 
abnormal  and  nl  is  normal;  AB  is  abnormal  and  NL  is  normal.  We  note  that  P(AB/ab)  + 
P(NL/ab)  =  1  and  that  PAB/nl)  +  P(NL/nl)  =  1 .  Thus,  of  the  four  conditional  probabilities,  two 
of  them  are  linearly  independent.  An  ROC  analysis  is  conducted  by  estimating  two  conditional 
probabilities,  based  upon  an  observation  variable  (Figure  2).  The  ROC  graph  (Figure  3)  is 
generated  by  moving  the  value  of  the  decision  threshold,  across  the  range  of  the  observational 
variable  Xp 
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Output 


Input 


AB 

NL 

ab 

P(AB/ab) 

True  positive 

P(NL/ab) 

False  negative 

nl 

P  (AB/nl) 

False  positive 

P(NL/nl) 

True  negative 

Figure  1 

Two-By-Two  Contingency  Table 
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P(X/AB) 


P(X/NL) 


Xc 

Observation  Variable 


Figure  2 

Model  of  ROC.  The  value  of  Xc  is  moved  across  the  range  of  the 
observation  variable,  generating  the  ROC  graph 
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Figure  3 
ROC  Graph 
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The  steps  in  conducting  an  ROC  analysis  are  well  known  (8-21),  The  2x2  contingency 
table  is  carefully  defined.  A  statistical  power  is  initially  selected  and  the  number  of  samples  and 
readers  are  calculated.  The  number  of  samples  (cases)  are  selected  so  that  one-third  are  very 
difficult  to  read,  one-third  are  very  hard  to  read,  and  one-third  are  fairly  hard  to  read.  A  5-point 
multiscale  reading  is  performed  on  each  case  by  each  reader  for  specific  image  features.  There 
are  a  number  of  available  ROC  software  programs  that  will  process  the  multiscale  reading  data  to 
provide  the  ROC  graph.  The  area  under  each  reader’s  ROC  graph  is  the  index  of  performance 
(the  probability  that  a  random  selected  patient  imdergoing  the  results  of  the  examination  will  be  a 
correct  true-positive). 

The  advantages  of  ROC  analysis  are  several.  It  is  a  well-known  tool,  often  used  by 
radiologists.  Software  exists  which  will  generate  the  ROC  graph  from  multi-scale  readings  by 
each  reader.  The  area  under  the  ROC  graph  is  used  as  an  index  of  reader  performance.  The 
disadvantages  of  ROC  analysis  is  that  it  remains  a  binary  decision  model.  It  is  difficult  to 
incorporate  population  statistics  (prior).  The  ROC  analysis  is  sensitive  to  modifying  the  readers 
or  cases. 

2.  Matrix  Modeling 

A  binary  examination  described  by  a  two-by-two  contingency  table  may  be  placed  into  a 
matrix  model  (Figure  4).  The  binary  decision  model  as  illustrated  has  two  input  states  (normal, 
abnormal)  and  two  output  states  (normal,  abnormal).  The  advantage  of  the  matrix  representation 
is  the  useful  manipulation  of  the  matrices  in  describing  probablistic  outcomes  of  multiple 
examinations.  For  example.  Figure  5  illustrates  the  results  of  cascading  two  examinations  while 
Figure  6  illustrates  a  decision  tree  model  of  the  two  cascaded  examinations.  The  results  of 
cascading  two  examinations  show  that  the  resultant  transition  probabilities  are  less  than  that  of 
each  individual  examination  -  a  result  sometimes  called  “leading  information.”  In  Figure  6,  the  a 
priori  probabilities  are  P(l)  and  P(2).  For  example  P(7/l)  is  given  by:  P(7/l)  =  P13P57  +  P14P67  = 


A  model  of  double  reading  is  illustrated  in  Figure  7.  In  double  reading,  reader  1  and 
reader  2  independently  read  the  results  of  die  examination.  They  then  compare  their  outcomes 
using  a  decision  rule.  Table  3  defines  the  set  of  possible  reader  outcomes  for  each  reader.  Tables 
4  and  5  provide  a  decision  rule  that  defines  the  conditions  under  which  the  readers  are  said  to  be 
equal  or  different.  A  more  conservative  decision  rule  would  be  to  say  that  the  two  readers  are 
equal  if  they  achieve  identical  outcomes  and  that  the  two  readers  are  different  if  they  do  not  have 
identical  outcomes.  It  is  believed  by  some  researchers  that  computer  aided  decision  algorithms 
could  serve  as  a  partial  second  reader. 
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Input  Output 

^11 


P„  =  P(AB/ab)  Pi2  =  P(NL/ab) 

P2,  =  P(AB/nl)  P22  =  P(NL/nl) 


Figure  4 

Matrix  Modeling 
of  an  Examination 
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(^13  ^57  +  ^14  ^6?) 
(^23  ^57  + ^24  ^67) 


(^13  ^58  +  ^14  ^es) 
(^23  ^58  +  ^24  ^68) 


Figure  5 

Modeling  of  Cascaded 
Examination 


Figure  6 

Decision  Tree  Modeling 
of  Cascaded  Examinations 
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Figure  7 

Model  of  Double  Reading 

8 


Table  3 

Reader  Outcome  for 
Modeling  of  Double  Reading 


Outcome  States 

Meaning  Assigned 

7  or  15 

Patient  recalled 

8  or  16 

Patient  negative  (return  1  year) 

9  or  17 

Benign  findings  (return  1  year) 

10  or  18 

Probably  benign  (return  6  month) 

11  or  19 

Suspicious  finding  (do  biopsy) 

12  or  20 

Malignant,  highly  suspicious  (do  biopsy) 

Table  4 

Decision  Rule  for 

Two  Readers  Outcomes  to  Be  Said  Equal 

Outcome: 

•  Negative  and  benign 

•  Suspicious  and  malignant 

•  Probably  benign 
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Table  5 

Decision  Rule  for  Two  Readers  Outcomes  to  Be  SaidDifferent 


Outcome: 

•  Benign  and  probably  benign 

•  Probably  benign  and  suspicious  finding 

•  Negative  and  suspicious  finding 

•  Benign  and  suspicious 

•  Negative  and  malignant 

•  Benign  and  malignant 

•  Probably  benign  and  malignant 
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The  sequence  of  outcomes  from  the  decision  rule,  based  upon  the  results  of  both  readers, 
can  be  assumed  to  be  an  n-sequence  of  the  symbol  set  (0,1)  where  the  alphabet  symbol  of  “0” 
means  equal  results  and  the  alphabet  symbol  of  “1 "  means  different  results  from  the  two  readers. 
If  these  results  are  observed  for  a  very  long  time,  the  sequence  of  symbols  (with  probability  one) 
will  be  “typical.”  That  is,  the  resultant  sequence  of  outcomes  from  the  decision  rule  is  ergodic. 
This  implies  that  a  n-sequence  of  outcomes,  as  n  increases,  will  result  in  typical  sequences  (those 
close  to  the  expected  value)  and  the  rest  being  non-typical  (those  with  probabilities  approaching 
zero  as  n  increases) 

3.  Uncertainty  Modeling 

Uncertainty  modeling  is  a  probablistic  scheme  (Figure  8)  in  which  there  is  a  sequence  of 
trials  (say  n)  and  conclusions  are  drawn  on  the  resultant  n-sequences  being  close  to  there 
expected  values  (9).  The  concept  of  typical  sequence,  as  introduced  by  Wolfowitz  (10),  who 
called  them  X  sequences,  and  as  used  by  Ash  (1 1),  is  related  to  the  asymptotic  behavior  of  its 
independent,  identically  distributed  random  variables  (12-15).  Let  Y  be  the  random  variable 
taking  on  the  values  of  (  y,  y2)(for  the  outcome  of  the  FBDM  test)  or  (y„  y2,y3,  y4,  (for  the 
outcome  of  the  readers  from  a  multiscale  reading,  respectively).  Suppose  that  the  experiment 
associated  with  Y  is  performed  “n”  times.  That  is ,  a  sequence  will  be  generated  of  Yj,  Y2, ...  Y„ 
of  independent,  identically  distributed  random  variables,  each  having  the  same  probability 
distribution  as  Y.  If  we  define  a  function  fj=fi(Y,,  Y2, ...,  Y„)  to  be  the  number  of  times  the 
symbol  y-,  occurs  in  the  sequence  Y„  Y2, ...  Y„,  then  y-,  will  f;  has  a  binomial  distribution  with 
parameters  n  and  Pj.  We  identify  a  “typical”  sequence  as  those  sequences  in  which  fj  is  “close”  to 
npj  for  every  i  where  np;  is  the  number  of  times  that  the  expected  value  of  y;  will  occur  in  the  n- 
sequence.  Then  we  note  that  the  set  of  nontypical  sequences  has  a  small  probability  of 
occurrence.  Thus  in  a  typical  sequence,  each  symbol  y^  occurs  approximately  with  its  expected 
frequency  of  np;.  The  formal  definition  is  the  following.  Given  c>0,  choose  any  positive  number 
k  such  that  1/k)^  <^/M  where  M  is  the  number  of  symbols  or  values  that  the  random  variable  Y 
takes  on.  Let  b  =  (b„  b2, ...,  bj  be  a  sequence  of  symbols,  each  bi  being  one  of  the  symbols  of 
the  set  (Y„  Yj, ...  YJ.  We  say  the  sequence  b  is  typical  if 

/fi(b)  -  npi /7(np,(l  p.)]|<k 

It  can  be  shown  that  the  following  is  true:  (a)  the  set  of  nontypical  sequences  of  length  “n”  has  a 
total  probability  <c;  and  (b)  the  number  of  typical  sequences  of  length  “n”  is  where  r„ 

approaches  zero  exponentially  as  n  grows  large  and  (c), 

H  =  (fi(b)log  p,  +  f2(b)  log  P2  +  ...+  fM(b)  log  p  J 
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X.  Input 


Y.  Output 


Xp  X25 . .  .j 


►  Y  Y  Y 

1  J,  i2, 


H(X)  H(X,Y)  H(Y) 

I(X;Y)  =  H(X)-H(XAO 

=  H(X)  =  H(Y)  -  H  (X,Y) 


Figure  8 

Uncertainty  Measures 


If  the  output  symbols  are  Xj,  X2, X^,  and  the  out  symbols  are  y„  y^, and  the 
transition  probabilities  are  [ajj],  where  =  P(yj/Xi),  i  =  1, M  and  j  =  1, L,  then  the  joint 
distribution  of  the  input  X  and  output  Y  is  given  by 

P(X  =  Xj,  Y  =  yj)  =  P(Xi)P(yj/Xi),  i  =  1, 2, M,  and  j  =  1, 2, L 
and  the  distribution  of  Y  is  given  by 

V(y)  =  P(X,)P(y/x,)  +  P  (x2)P(y/x2)  + ...  +P(xM)P(y/xM). 

For  M=2  and  L=5,  then  P(Y  =  yj)  =  P(xi)P(yj/x,)  +  PCxjjPCy/xj),  j  =  1, ...,  5.  If  the  experiment  is 
repeated  “n”  times,  then  if  b  =  (b„  bj, ...,  b  j  is  a  sequence  of  resultant  n-symbols,  each  bj  being 
one  of  the  elements  (y„  yj, ...,  ys),  then  we  say  that  the  n-sequence  is  typieal  if  it  satisfies  the 
above  definition. 

The  significance  of  the  use  of  typical  sequences  is  in  analyzing  the  model  of  the 
diagnostic  test  and  reader  responses.  Typical  sequences  reduce  the  number  of  n-sequences  that 
require  analysis.  For  a  large  n  (number  of  independent  trials),  there  are  approximately  2*"^^ 
typical  input  n-sequences,  each  with  probability  roughly  Similarly  ther  are  2"”^^  typical 

output  n-sequences  and  2"”^  '^  typical  pairs  of  input  and  output  n-sequences.  A  typical  pair  may 
be  generated  by  first  selecting  a  typical  output  n-sequence  y  and  then  selecting  a  typical  input  n- 
sequence  x  such  that  (x,y)  is  a  typical  pair.  Since  the  number  of  typical  output  sequences  is  2"“^^^ 
and  the  number  of  typical  pairs  is  then  for  each  typical  output  sequence  y,  there  are 

2nH(X,Y)-H(Y)  _  2nH(X/Y) 
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input  sequences  x  such  at  (x,y)  is  a  typical  pair.  That  is,  if  a  typical  sequence  y  is  presented  then 
the  number  of  typical  input  sequences  possible  are  approximately  2""^^  (not  each  with 
approximately  the  same  probablity,  2  "”^^.  This  is  important  since  it  reduces  the  number  of  n- 
sequences  that  are  to  be  analyzed.  Hence,  if  the  experiment  is  conducted  say  100  times  (n=l  00 
or  reading  out  100  cases)  then  we  can  expect  approximately  2“^”^^  typical  sequences,  each  with 
approximate  probability  of  2*'“”^.  It  is  known  that,  given  the  transition  probabilities  (such  as 
obtained  from  an  ROC  analysis),  then  by  varying  the  input  distribution  P(X),  one  can  obtain  a 
mavimiim  of  the  mutual  information,  I(X;  Y)  =  H(X)  +  (H(Y)  -  H(XA^),  between  the  input,  X, 
and  the  output,  Y,  which  is  called  the  channel  capacity,  C.  Hence,  given  the  transition 
probabilities  between  input  and  output,  (Py),  there  exists  an  input  distribution  P(X)  such  that  the 
mutual  information  between  X  and  Y,  I(X;  Y),  is  a  maximum.  The  result  suggests  that  for  a 
given  input  distribution  P(X),  the  a  priori  distribution,  the  transition  probabilities  may  be 
changed  to  better  approach  the  channel  capacity,  C. 

If  we  are  modeling  an  examination,  then  Figure  9  illustrates  the  relations  between 
different  uncertainty  measures.  These  measures  are  given  by: 

H(X)  =  -£PilogPi 
H(Y)  =  -EPjlogPj 
and 

H(X,Y)  =  -Ei  Ej  P(Xi,  Yj)  log  P  (Xi,  Yi) 

The  mutual  information  between  (X,Y)  is  given  by 

I(X;Y)  =  H(X)  -  UQUY) 

=  L  Ej  P(Xi,  Yj)  log  P(Xi,  Yi) 

P(Xi) 

=Ii  Ej  P(Xi,  Y^)  log  P(Xi,Yi) 

P(Xi)  P  (Y) 


Figure  9  illustrates  a  graphical  schematic  of  these  relationships. 

Figure  10  is  a  model  of  double  reading  as  previously  cited.  We  are  interested  in  the 
mutual  information,  I  (A;  B,  C,  D,  E).  This  is  the  mutual  information  regarding  “A”  on  condition 
of  the  occurrence  of  (B,  C,  D,  E).  It  is  given  by: 

I  (A;  B,  C,  D,  E)  =  I  (A/C)  + 1  (A;  B/C) 

+  I(A;D/B)I(A;D/C) 

+  I(A;E/B)  +  I(A;E/C) 

+ 1  (A;  E/D). 

The  conditional  mutual  information  terms,  such  as  I  (A;  D/C),  is  the  average  amount  of 
information  about  the  set  “A”  provided  by  an  observation  from  the  set  “D”  after  an  observation 
from  the  set  “C”. 
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Figure  9 

Uncertainty  Measures 
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Figure  1 1  is  a  model  for  display  protocols  (either  laser  printed  film  or  soft  copy,  the 
grayscale  workstation)  for  digital  mammography.  Then  the  occurrence  of  (B,  C,  D,  E,  F,  G)  that 
tells  us  about  the  set  “A”  is  the  following: 

I  (A;  B,  C,  D,  E,  F,  G)  =  I  (A;  C)  + 1  (A;  B/C) 

+ 1  (A;  D/B)  + 1  (A;  D/C)  +  J  (A;  E/B) 

+ 1  (A;  E/C)  + 1  (A;  E/D)  + 1  (A;  F/B) 

+ 1  (A;  F/C)  + 1  (A;  F/D)  + 1  (A;  F/E) 

+ 1  (A;  G/B)  + 1  (A;  G/C)  + 1  (A;  G/D) 

+  I(A;G/E)  +  I(A;G/F). 

The  necessary  parameters  for  this  model  are  conditional  probabilities.  The  value  of  I  (A; 
B,C,D,E,F,G)  tells  us  the  number  of  typical  sequences  in  an  n-sequence  as  well  as  their 
probability  of  occurrence.  The  decomposition  of  I  (A;  B,  C,  D,  E,  F,  G)  into  simpler  terms,  such 
as  I  (A;  G/F)  provides  for  a  much  simpler  model. 

We  have  extended  our  mammography  workflow  modeling  to  include  specific  tasks  as 
seen  at  several  sites  (Figures  12-14).  Figure  15  illustrates  the  types  and  amounts  of  resources 
used  at  each  step  of  the  patient  workflow  tasks.  Figure  16  illustrates  the  tasks  and  sequence  in 
which  they  occur. 
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The  screening  mammography  workflow  is  dependent  upon  the  technology  that  is  being  utilized  -  screen- 
film,  digital,  or  both.  The  nature  of  the  setting  -  private  or  academic  -  will  also  impact  the  workflow. 

SCREEN-FILM  MAMMOGRAPHY  WINDOW 


Figure  12.  Screen-Film  Screening  Steps 


DIGITAL  MAMMOGRAPHY  WORKFLOW 


Figure  13.  Digital  Screening  Steps 


SCREEN-FILM  TO  DIGITAL  MAMMOGRAPHY  WORKFLOW 


Figure  14.  Screening  Steps  for  Settings  Transitioning  from  Screen-Film  to  Digital 
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_ DESCRIPTION _ 

Schedule  appointment  _ 

Register  patient _ 

Retrieve/create  jacket _ 

Prepare  mammography  unit  and  dressing 
room _ 

Escort  patient  to  dressing  room _ 

Allow  patient  to  undress _ 

Acquire  images _ 

Process  film _ 

Assess  images  for  quality _ 

Mount  past  and  present  films _ 

Read  images _ _ _ 

Allow  patient  to  dress _ 

Consult  with  patient _ 

Dictate  report _ 

Transcribe  report _ 

Review  and  sign  report  _ 


THROUGHPUT  ANALYSIS 


The  three  techniques  for  performance  measurement  of  a  system  are  analytical  modeling, 
simulation,  and  measurement  of  an  actual  system  (16).  The  major  consideration  in  deciding 
which  evaluation  technique  to  use  is  the  life-cycle  stage  in  which  the  system  resides. 
Measurements  are  possible  if  technology  similar  to  the  system  being  evaluated  already  exists.  If 
the  system  under  study  is  new,  then  only  analytical  modeling  and  simulation  methods  are 
possible.  Simulation  is  difficult  and  time-consuming  to  accomplish.  For  the  mammography 
systems,  the  only  reasonable  evaluation  technique  is  analytical  modeling.  The  performance 
metrics  being  used  in  our  modeling  of  these  systems  are  throughput  and  diagnostic  accuracy. 
Throughput  is  defined  as  the  rate  (jobs-per-minute)  at  which  requests  can  be  serviced  by  the 
system.  The  throughput  of  a  system  generally  increases  as  the  load  on  the  system  initially 
increases,  without  the  addition  of  more  resources;  after  a  certain  load,  the  throughput  stops 
increasing  due  to  one  or  more  bottlenecks.  Throughput  analysis  is  accomplished  by  listing  the 
resources  used  and  the  steps  required  to  complete  one  job.  The  resource  with  the  highest  total 
service  demand  has  the  highest  utilization  and  is  called  the  bottleneck  resource. 

Table  1  and  Table  2  (17)  are  the  resource  utilization  tables  generated  for  two  throughput 
and  bottleneck  analyses.  The  top  row  of  each  table  lists  the  resources  being  modeled.  The  rows 
of  tables  are  die  steps  required  to  complete  one  reading.  The  bottom  rows  (throughput  per 
minute)  are  generated  by  alternately  assuming  that  each  resource  in  the  columns  is  operating  as 
rapidly  as  it  possibly  can. 

The  bottleneck  analysis  described  above  is  based  on  Little's  Law  (18)  of  mean  value 
queuing  analysis.  Littie's  Law  states  that  the  mean  number  of  jobs  in  a  system  equals  the  mean 
throughput  rate  multiplied  by  the  mean  time  in  the  system.  We  have  conducted  similar  studies 
for  screen-film  mammography.  Table  1  (labeled  “MCV”)  is  a  Resource  Utilization  table  for 
mammography  at  the  Medical  College  of  Virginia  (provided  by  Dr.  Ellen  Shaw  de  Paredes)  and 
Table  2  (labeled  "SP")  is  for  an  outpatient  Breast  Imaging  Center. 

The  throughput  analysis  is  conducted  by  obtaining  the  upper  and  lower  bound  (Figure  17) 
on  the  actual  throughput  as  a  function  of  one  or  more  of  the  resources.  Upper  and  lower  bounds 
are  required  to  understand  the  otherwise  complex  analysis  of  the  actual  throughput.  If  the 
radiologist  is  selected  as  the  independent  variable  (increasing  the  number  of  radiologists),  then 
the  upper  bound  on  the  throughput  is  given  by 

Upper  Bound  =  min[  IT^y^,  N/(T5ys  + 

where  T^y^  is  the  time  used  by  the  system  less  the  radiologist  and  resident  time  and  T,ea<i„  is  the 
time  used  by  the  radiologist  and  the  resident  reading  out  the  cases  (while  system  is  waiting). 

Then  l/T^yj  is  the  maximum  throughput.  The  term  N/(T,y,  +  T^^ader)  relating  increases  in  the 
number  of  radiologists  (and  residents)  increases  linearly  with  N  until  it  reaches  l/T^y^,  after  which 
the  system  cannot  increase  its  throughput  no  matter  how  many  radiologists  (and  residents)  are 
added  to  the  reading  room.  This  is  the  case  when  the  radiologist  is  unable  to  do  any  other  work 
above  the  throughput  limit  because  the  system  is  then  queuing  examinations.  We  note  that  as  N 
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increases  (N  could  be  other  resources  such  as  technologists),  then  there  is  a  method  for 
calculating  the  lower  bound  on  the  throughput.  That  is, 

Lower  Bound  of  Throughput  =  N/(NTsys  +  T^j„). 

This  will  occur  when  the  system  is  busy  doing  other  things  and  the  newest  case  is  last  in  the 
queue  to  receive  service.  Figure  17  illustrates  this  analysis  of  throughput. 
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Number  of  Jobs 

^^(Treader  +  Tjystem)^ 


system 


Figure  17 

Throughput  Analysis 
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SUMMARY 


We  have  reviewed  modeling  of  mammography  examinations.  Modeling  is  essential  due 
to  the  large  number  of  variables.  However,  a  carefUlly  selected  model  will  require  only  the 
estimation  of  the  model  parameters.  Of  the  models  we  have  described,  the  uncertainty  modeling 
is  superior  owing  to  its  restrained  relationships  and  requiring  only  the  estimation  of  conditional 
probabilities.  The  throughput  analysis  is  a  method  that  is  simple  to  utilize  in  estimating 
workflow. 

CONCLUSIONS 

1 .  Reading  of  the  screen-films  is  far  simpler  than  the  soft-copy  reading  of  the  digitized 
images. 

This  is  largely  due  to  the  lack  of  user  friendly  interface  systems  on  the  soft-copy  display. 
It  is  also  due  to  the  increased  time  required  to  read  soft-copy  displays. 

2.  For  those  ROC’s  thus  far  analyzed,  there  is  approximately  a  8  to  10%  decrease  in 
diagnostic  reading  of  soft-copy  displays. 

Since  the  readers  did  well  on  the  analog  screen-film  examinations,  it  is  likely  that  the 
problem  resides  in  the  proper  display  protocols  for  the  soft-copy  displays 

The  display  protocols  on  the  soft-copy  display  require  7  to  12  minutes  to  read  a  case. 

LAY  PUBLIC  ABSTRACT 

Telemammography  is  the  transmission  and  interpretation  of  digitized  mammographic 
images.  The  use  of  telemammography  will  allow  for  interpretation  of  on-site  mammograms 
from  distant  areas  by  expert  radiologists  at  a  central  site.  This  can  potentially  increase  the  access 
to  mammography  for  many  women. 

We  are  testing  a  telemammography  system  utilizing  digitized  mammograms  that  are 
transmitted  through  telephone  lines  or  satellites  and  are  interpreted  on  computer  workstations. 
The  goal  of  this  study  is  to  determine  the  requirements  to  deliver  high  quality  mammographic 
images  from  remote  locations. 

To  test  the  system  we  have  collected  400  normal  and  abnormal  mammograms  which  have 
been  interpreted  by  12  radiologists.  The  radiologists  are  interpreting  the  studies  on  film  as  well 
as  computer  workstations,  and  we  are  comparing  their  accuracy  with  two  methods. 
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We  also  are  assessing  the  performance  of  the  system  by  comparing  the  patient  workflow 
when  their  mammograms  are  performed  using  routine  mammographic  film  versus  digitized 
mammograms. 

We  have  found  that  reading  mammograms  on  the  workstations  versus  on  film  reduces  the 
patient  throughput.  Also  because  of  the  decreased  brightness  of  images  on  the  workstations 
versus  viewing  films  on  traditional  light  boxes,  the  display  of  some  images  is  difficult. 
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APPENDIX  I 


List  of  Mammography  Screen  Film  Cases  by  Case  Number, 
Pathology,  and  the  Expert  Panel  Readings  for  Diagnosis, 
Microcalcifications,  Mass,  and  Focal  Areas  of  Asymmetry 


PATHOLOGY  REPORT 

Case  Group  Diagnosis  Micro-CalciHcations  Mass  Focal  area  of  Asymmentry 

#  or  Architectural  Distortion 


PATHOLOGY  REPORT 

Case  Group  Diagnosis  Micro-CalciHcations  Mass  Focal  area  of  Asymmentry 

#  or  Architectural  Distortion 


Malignant 


PATHOLOGY  REPORT 

Case  Group  Diagnosis  Micro-CalciHcations  Mass  Focal  area  of  Asymmentry 

#  or  Architectural  Distortion 
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PiagDflsls 

CO 

CO 

CO 

CO 

CO 

CD 

CO 

CO 

CO 

CO 

CO 

CO 

CO 

CO 

CO 

CD 

s 

CO 

CO 

CO 

CO 

CO 

CO 

CO 

U. 

Normal: 

Abnormal: 

FAD 

z 

z 

z 

z 

z 

< 

Z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

< 
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z 

z 

Z 

z 

z 

z 

z 

z 

z 

< 

z 

LU 
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z 
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z 

z 

z 

z 

z 
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< 

z 

Z 
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z 

z 

z 

Z 

z 

O 
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z 

< 

z 

z 

z 

z 

z 

z 

z 

< 

z 

|z 

z 

z 

z 
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z 

z 
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Z 

z 

z 

z 

z 

z 

z 

z 

i 

<: 

z 

z 

Case 

Number 

s 

lO 

o 

CO 

o 

T— 

N. 

O 

T— 

(N 

CO 

■"it 

T— 

to 

CO 

T- 

T“ 

T— 

00 

T— 

C3> 

T— 

T— 

§ 

eg 

eg 

▼" 

CO 

eg 

T— 

S 

LO 

eg 

T— 

CD 

eg 

OO 

eg 

<3) 

CM 

O 

CO 

T— 

s 

eg 

CO 

T- 

CO 

CO 

T— 

2 

CO 

CO 

N. 

CO 

00 
CO : 

T-  ! 

(3> 

CO 

5 

- 

s 

CO 

a> 

o 

00 

o 

a> 

o 

100 

101 

CM 

O 

CO 

o 

T— 

104 

105 

106 

107 

108 

o> 

o 

110 

T— 

112 

CO 

r“ 

T— 

in 

CO 

T“ 

118 

119 

120 

121 

eg 

CM 

123 

124 

125 
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X 


o 

Benign: 

Malignant: 

Diagnosis 

IS 

CO 

s 

CD 

CD 

CD 

CD 

S 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

CD 

S 

CD 

CD 

CD 

CD 

CD 

CD 

UL 

Normal: 

Abnormal: 

FAD 

z 

z 

z 

Z 

Z 

Z 

z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

< 

Z 

< 

z 

z 

z 

Z 

z 

Z 

Z 

Z 

Z 

z 

Z 

z 

z 

LU 

Normal: 

Abnormal: 

uCa 

z 

z 

< 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

< 

z 

z 

Z 

z 

< 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

Q 
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Abnormal: 
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< 

z 

< 

z 

z 

< 

z 

< 

z 

z 

z 

z 

z 

z 

Z 

z 

z 

z 

j 
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< 

z 

z 

z 

< 

:Z 

z 

z 

< 

z 

i 

z 

< 

Case 

Number 

142 

CO 

Wl 

CO 

149 

o 

in 

In 

CM 

in 

T“ 

£91. 

154 

991. 

r^ 

in 

158 

691. 

CO 

s 

991. 

CO 

CO 

h- 

CO 

69 1. 

ozt 

T— 

T— 

CM 

N. 

CO 

1^ 

175 

T“ 

1^ 

081. 

00 

CM 

00 

CO 

00 

- 

1261 

00 

CM 

129 

130 

131 

132 

133 

134 

135 

136 

N. 

CO 

138 

139 

140 

141 

142 

143 

144 

145 

146 

■r- 

148 

149 

150 

151 

CM 

in 

153 

s 

155 

156 
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O’E  &  ^  comoQSScQmcQScQS 
rS=5.2 


cQ  m  m 


03  CO  CO  CD  CO  ^  m  CD 


Z  Z  Z  Z 


z  z  z  z  z  z  z  z  z  z  z  z  z  z  z  z 


^  (0 
CO  c 

tn  i  I 

o  c 
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^  CO 
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o  i  i 
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<»00  00  00  00  Q00>050)0>0^0)0)0)0>OOOOOOt-t-v-t“t—  t-x“t-CVJCNi 
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X 

T“ 

T— 

T“ 

o 

Benign: 

Malignant: 

Diagnosis 

00 

00 

00 

00 

00 

00 

00 

00 

00 

OO 

00 

00 

00 

00 

00 

00 

00 

00 

00 

00 

I 

Normal: 

Abnormal: 

FAD 

z 

z 

z 

z 

z 

z 

z 

< 

z 

z 
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< 

z 

z 

Z 

z 

z 

z 

z 

z 

z 

z 

z 

Z 

z 

z 

z 

z 

z 

z 

z 
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Abnormal: 
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z 

z 

z 

z 

z 

< 

z 

z 

z 

z 

z 

z 

z 

z 

< 

< 

z 

z 

z 

z 

z 

z 

< 

< 

< 

z 

z 

z 

z 

z 

< 

Q 

Normal: 

Abnormal: 

Mass 

z 

z 

z 

^z 

z 

z 

z 

z 

< 

z 

< 

z 

z 

z 

z 

z 

z 

z 

z 

< 

< 

z 

z 

z 

z 

< 

z 

< 

z 

z 

z 

I 

Case 

Number 

CO 

CM 

CM 

CM 

to 

CM 

CM 

CO 

L7Z 

833 

CJ> 

CM 

CM 

o 

CO 

CM 

CM 

CM 

CO 

CM 

233 

CM 

lO 

CO 

CM 

CO 

CO 

CM 

h- 

CO 

CM 

00 

CO 

CM 

C3) 

CO 

CM 

o 

CM 

s 

CO 

245 

C3> 

o 

m 

CM 

T— 

in 

CM 

CM 

in 

CM 

CO 

in 

CM 

in 

in 

CM 

CO 

in 

CM 

in 

CM 

1 

T— 

00 ! 
00 

1^ 

190 

191 

192 

193 

194 

195 

196 

197 

00 

05 

199 

200 

201 

202 

CO 

o 

CM 

204 

205 

CO 

O 

CM 

207 

00 

o 

CM 

C3) 

O 

CM 

210 

211 

CM 

213 

214 

in 

CM 

i 

217 

218 

Page  7  of  12 


Typical  Reader  Response 


X 

T— 

T— 

t— 

T— 

T- 

T— 

o 

Benign: 

Malignant: 

Diagnosis 

m 

CO 

CO 

CO 

CO 

CO 

CO 

CO 

s 

CO 

"S. 

CO 

CO 

CO 

CO 

00 

CO 

00 

'S. 

00 

00 

00 

00 

00 

I 

Normal: 

Abnormal: 

FAD 

z 

z 

z 

z 

z 

z 

z 

z 

< 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

LU 

Normal: 
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z 
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z 
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z 
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z 
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z 

z 

z 

z 

z 

z 

z 

^z 

z 

z 

z 

z 

z 

z 

z 

< 

z 

z 

z 

z 

< 

Case 

Number 

00 

m 

CM 

69Z 

260 

CM 

CD 

CM 

CO 

CO 

CM 

in 

CO 

CM 

CO 

CO 

CM 

CO 

CM 

00 

CO 

CM 

o> ' 

CO 

CM 

o 

CM 

CM 

ZLZ 

CO 

CM 

CM 

CO 

CM 

r- 

CM 

00 

CM 

cn 

h- 

CM 

o 

00 

CM 

T— 

00 

CM 

CM 

oo 

CM 

CO 

00 

CM 

s 

CM 

to 

00 

CM 

00 

CM 

287 

00 

00 

CM 

C35 

00 

CM 

T— 

05 

CM 

292 

- 

219 

o 

CM 

CM 

221 

222 

CM 

CM 

224 

225 

226 

227 

228 

229 

o ' 

CO 

CM 

231 

CM; 

CO 

CM 

233 

234 

235 

<5" 

CO 

CM 

237 

00 

CO 

CM 

239 

240 

241 

242 

243 

244 

245 

246 

247 

248 

249 
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X 

T“ 

X— 

T- 

-r- 

T- 

T- 

T— 

T— 

r- 

T— 

o 

Benign: 

Malignant: 

Diagnosis 

OQ 

m 

CO 

GQ 

CQ 

S' 

CD 

GQ 

S 

CD 

CD 

s 

CD 

S 

CD 

GQ 

S 

GQ 

CD 

GQ 

GQ 

s 

S 

GQ 

GQ 

GQ 

CD 

GQ 

GQ 

GQ 

I 

Normal: 

Abnormal: 

FAD 

z 

Z 

z 

z 

Z 

Z 

z 

z 

Z 

Z 

Z 

Z 

Z 

z 

< 

z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

Z 

LU 

Normal: 

Abnormal: 

uCa 
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z 
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z 

z 

z 

z 

z 

< 

z 

z 

z 

z 

< 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

< 

z 

z 

< 

Case 

Number 

293 

294 

295 

296 

297 

298 

T— 

o 

CO 

CM 

o 

CO 

CO 

o 

CO 

304 

305 

307 

309 

310 

313 

315 

316 

318 

319 

320 

321 

CM 

CM 

CO 

CO 

CM 

CO 

CO 

CM 

CO 

327 

328 

329 

o 

CO 

CO 

CO 

CO 

CM 

CO 

CO 

T— 

o 

lO 

CM 

s 

CM 

252 

253 

in 

CM 

255 

"CD 

in 

CM 

in 

CM 

258 

259 

260 

T” 

CO 

iCM 

CM 

CO 

CM 

263 

CO 

CM 

in 

CO 

CM 

266 

267 

I 

c:> 

CO 

CM 

270 

CM 

272 

CO 

1^ 

CM 

CM 

in 

CM 

CO 

CM 

r^ 

CM 

00 

CM 

ay 

CM 

li 
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GQ 

GQ 
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GQ 

GQ 

GQ 

CD 

GQ 

GQ 
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|z 

< 

Case 

Number 

CO 

CO 

CO 

s 

CO 

h- 

CO 

CO 

00 

CO 

CO 

C5) 

CO 

CO 

o  1 

CM 

CO 

CO 

s 

CO 

<y> 

CO 

CO 

lO 

O) 

CO 

h- 

CO 

O) 

C3> 

CO 

o 

o 

o 

CM 

O 

s 

h- 

o 

00 

o 

O) 

o 

o 

5 

CM 

T— 

CO 

T— 

LO 

T- 

CO 

5 

00 

V- 

o> 

T— 

CM 

CM 

CM 

CO 

CM 

- 

1 

CO 

00 

CM 

285 

K 

oo 

CM 

00 

00 

CM 

O) 

00 

CN 

o ! 
O) 
CM 

291 

292 

293 

294 

295 

296 

297 

00* 

a> 

CM 

C5) 

O) 

CM 

o' 

o 

CO 

301 

It 

CO 

o 

CO 

304 

To" 

It 

306 

307 

00 

O 

CO 

309 

310 
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CO 
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o 

CM 

CO 
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00 
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T— 

CM 

CO 

lO 
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a> 

CM 

CM 

CM 

CM 

CO 

CO 

CO 

CO  CO 

CO 

CO 

CO 

CO 

CO 

Tt 

lO 

lO 

lO 
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lO 

lO 

m 

lO 

'M* 

Tj- 

Tt  Tj- 

Tt 

I 
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APPENDIX  n 


Original  screen-film  mammography  image  compared  to  laser 
film  printed  image  of  laser  digitized  image. 


Reference  to  films: 

Non-Digitized  (Analog) 

Removed  from  patient  x-ray  jacket 
#1  =  Original 
#2  =  Original 
#3  =  Original 


Now  Have: 

Copies  of  Digital  Images  that  were  digitized  from  originals  (Analog)  #1,  #2,  and  #3 


12  bit  images 

#4  =  digital  image  of  original  #1  by  2180  H  Kodak  Laser  Printer 
#5  =  digital  image  of  original  #2  by  2180  H  Kodak  Laser  Printer 
#6  =  digital  image  of  original  #3  by  2180  H  Kodak  Laser  Printer 

8  bit  images 

#7  =  digital  image  of  original  #1  by  AGFA  5200  Laser  Printer 
#8  =  digital  image  of  original  #2  by  AGFA  5200  Laser  Printer 
#9  =  digital  image  of  original  #3  by  AGFA  5200  Laser  Printer 


Note:  Numbers:  1, 2,  and  3  indicate  non-digitized  (Analog)  images  that  were  removed  from 
patient’s  original  x-ray  film  folders  (jackets) 


Note:  Numbers:  4,  5,  6,  6,  8,  and  9  indicate  images  made  from  the  digital  images  of  #  1,  2, 
and  3 


APPENDIX  III 


ROC  Analysis 


Results  for 
Reader  #2 
Reader  #3 
Reader  #4 
Reader  #5 
Reader  #6 
Reader  #7 
Reader  #8 
Reader  #9 
Reader  #11 
Reader  #14 


39 


Reader  Number 


Reader  List 
Screen-Films  Read 


Digital  Soft  Copy  Read 

297 

0 


2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 


382 

329 
398 
326 
384 

330 
330 
330 
229 
268 
069 
098 
277 


373 

0 

180 

0 

0 

0 

0 

84 

0 

0 

362 


15 


0 


91 


DISPLAY  PROTOCOLS 


Display  A 


Display  B 


Display  C 


Display  D 


Reader02  Digital 

Totals  Type  Mass  Calc  Fad  Diag 

Total 

N^al  144  241  216  270  215 

Total 


Reader02  Analog 

Totals  Type  Mass  Calc  FAD  Diag 

Total 

N^al  144  241  216  270  215 

Total 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

~~  n  DATE  - 


10/21/1997 


TIME  ” 


15:12:32 


R  0  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  02  -  ROC  for  Mass  -  Analog 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  201. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  198. 

RESPONSE  DATA: 

CATEGORY  1234 

ACTUALLY  NEGATIVE  CASES  51.  60.  73.  5. 

ACTUALLY  POSITIVE  CASES  23.  44.  58.  20. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .060  .085  .448  .746  1.000 

TPF:  .000  .268  .369  .662  .884  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .6248  B=  .7673 

Z(K)=  -.662  .131  1,375  1.558 

LOGL-  -578.5890 
GOODNESS  OF  FIT  — 

CHI  SQUARE-  17.0981  WITH  2  DEGREES  OF  FREEDOM,  P=  .0002 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .6203  B-  .7865 

Z(K)=  -.683  .181  1.276  1.577 

LOGL-  -574.2211 
GOODNESS  OF  FIT  -- 

CHI  SQUARE-  5.5634  WITH  2  DEGREES  OF  FREEDOM,  P-  .0619 
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A 

B 

Z(  1) 
Z(  2) 
Z(  3) 
Z(  4) 


CORRELATION  MATRIX: 
1.0000  .3041  .4910 

.3041  1.0000  .3422 

.4910  .3422  1.0000 

.5166  .0461  .5999 

.3380  -.4639  .1835 

.2696  -.5589  .0986 


.5166  .3380  .2696 

.0461  -.4639  -.5589 

.5999  .1835  .0986 

1.0000  .5124  .4170 

.5124  1.0000  .8862 

.4170  .8862  1.0000 


AREA  =  .6871  STD.  DEV. (AREA)  -  .0274 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF 


(LOWER  BOUND,  UPPER  BOUND) 


.005 

.0799 

.010 

.1132 

.020 

.1598 

.030 

.1951 

.040 

.2245 

.050 

.2503 

.060 

.2733 

.070 

.2944 

.080 

.3138 

.090 

.3320 

.100 

.3491 

.110 

.3652 

.120 

.3806 

.130 

.3952 

.140 

.4093 

.150 

.4227 

.200 

.4834 

.250 

.5359 

.300 

.5825 

.400 

.  6633 

.500 

.7325 

.  600 

.7937 

.700 

.  8491 

.800 

.  9001 

.900 

.9483 

.  950 

.  9722 

( 

( 

.0363 

t 

.1546 

) 

.0587 

! 

.1966 

) 

( 

.0940 

r 

.2501 

) 

( 

.1231 

t 

.2880 

) 

( 

.1487 

t 

.3185 

) 

( 

.1719 

f 

.3443 

) 

( 

.1932 

f 

.3671 

) 

( 

.2131 

t 

.3876 

) 

( 

.2318 

t 

.4063 

) 

( 

( 

.2495 

f 

.4236 

) 

.2663 

t 

.4398 

) 

( 

.2823 

r 

.4550 

) 

( 

.2977 

r 

.4694 

) 

( 

.3125 

f 

.4830 

) 

( 

.3267 

t 

.4961 

) 

( 

.3404 

f 

.5086 

) 

( 

.4029 

f 

.5646 

) 

( 

.4574 

t 

.6130 

) 

( 

.5060 

f 

.  6559 

) 

( 

.5901 

t 

.7307 

) 

( 

.  6622 

r 

.7945 

) 

( 

.7265 

t 

.  8499 

) 

( 

.7861 

r 

.8983 

) 

( 

.8439 

t 

.  9399 

) 

( 

.  9040 

r 

.9745 

) 

( 

.9387 

t 

.  9888 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 

LOWER 
(  FPF  , 

BOUND 

TPF  ) 

UPPER 
(  FPF  , 

BOUND 
TPF  ) 

(  .0573, 

(  .1010, 

.2675) 

.3508) 

(  .0332, 

(  .0670, 

.2050) 

.2884) 

(  .0936, 

(  .1460, 

.3384) 

.4175) 
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Reader  02  Digital  ROC:  Type=  Mass  N=200  Areas.OSOS  STD  Area=.0417 


False  Alarms 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  man? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  cuarve? 

(y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


10:39:20 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  02  Digital  Mass  resp  for  296  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  241. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  55. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  71.  94.  47.  9.  20. 

ACTUALLY  POSITIVE  CASES  4.  3.  3.  4.  41. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .083  .120  .315  .705  1.000 

TPF:  .000  .745  .818  .873  .927  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2792  B=  .3812 

Z{K)=  -.540  .480  1.173  1.385 

LOGL=  -383.9077 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.2917  B=  .4382 

Z(K)=  -.536  .474  1.152  1.412 

LOGL=  -383.4332 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 

A  .0461  .0148  .0023  .0021  .0018  .0015 
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B 

.0148 

.0159 

.0011 

.0000 

-.0017 

-.0031 

Z( 

1) 

.0023 

.0011 

.0072 

.0032 

.0022 

.0018 

z( 

2) 

.0021 

.0000 

.0032 

.0070 

.0049 

.0044 

z( 

3) 

.0018 

-.0017 

.0022 

.0049 

.0105 

.0096 

z{ 

4) 

.0015 

-.0031 

.0018 

.0044 

.0096 

.0137 

CORREIiATION  MATRIX: 

A 

1.0000 

.5450 

.1262 

.1148 

.0832 

.0600 

B 

.5450 

1.0000 

.1012 

.0038 

- .1314 

-.2086 

z{ 

1) 

.1262 

.1012 

1.0000 

.4529 

.2472 

.1861 

z{ 

2) 

.1148 

.0038 

.4529 

1.0000 

.5694 

.4491 

z{ 

3) 

.0832 

-.1314 

.2472 

.5694 

1.0000 

,8001 

z( 

4) 

.0600 

-.2086 

.1861 

.4491 

.8001 

1.0000 

AREA  =  .8816  STD.  DEV.  (AREA)  =  .0343 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.5647 

( 

.3535  , 

.7586 

) 

.010 

.6073 

( 

.4123  , 

.7782 

) 

.020 

.6523 

( 

.4770  , 

.7998 

) 

.030 

.6799 

( 

.5172  , 

.8137 

) 

.040 

.7000 

( 

.5466  , 

.8243 

) 

.050 

.7159 

( 

.5698  , 

,8329 

) 

.060 

.7292 

( 

.5890  , 

.8403 

) 

.070 

.7405 

( 

.6052  , 

.8468 

) 

.080 

.7505 

( 

.6194  , 

.8527 

) 

.090 

.7593 

( 

.6318  , 

.8580 

) 

.100 

.7673 

( 

.6430  , 

.8630 

) 

.110 

.7746 

( 

.6531  , 

.8675 

) 

.120 

.7814 

( 

.6622  , 

.8718 

) 

.130 

.7876 

( 

.6707  , 

.8759 

) 

.140 

.7934 

( 

.6784  , 

.8797 

) 

.150 

.7989 

( 

.6857  , 

.8833 

) 

.200 

.8220 

( 

.7154  , 

.8991 

) 

.250 

.8405 

( 

.7382  , 

.9123 

) 

.300 

.8559 

( 

.7565  , 

.9235 

) 

.400 

.8812 

( 

.7853  , 

.9420 

) 

.500 

.9018 

( 

.8080  , 

.9566 

) 

.600 

.9196 

( 

.8275  , 

.9686 

) 

.700 

.9359 

( 

.8456  , 

.9786 

) 

.800 

.9516 

( 

.8640  , 

.9869 

) 

.900 

.9681 

( 

.8855  , 

.9939 

) 

.950 

.9779 

( 

.9006  , 

.9969 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
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CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 


( 

FPF  , 

TPF  ) 

( 

( 

.0790, 

.7495) 

( 

( 

.1248, 

.7844) 

( 

( 

.3177, 

.8608) 

( 

( 

.7039, 

.9365) 

( 

LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

.0504, 

.7165) 

{  .1184, 

.7803) 

.0881, 

.7578) 

(  .1709, 

.8093) 

.2619, 

.8443) 

(  .3781, 

.8761) 

.6439, 

.9269) 

(  .7587, 

.9451) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


10:39:20 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  02  Analog  Calc  resp  for  296  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  216. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  80. 

RESPONSE  DATA: 

CATEGORY  1  2  3  4  5 

ACTUALLY  NEGATIVE  CASES  173.  22.  12.  3.  6. 

ACTUALLY  POSITIVE  CASES  8.  7.  4.  11.  50. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .028  .042  .097  .199  1.000 

TPF:  .000  .625  .762  .813  .900  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.9685  B=  .8071 

Z(K)=  .845  1.298  1.732  1.915 

LOGL=  -255.8930 

CHI-SQUARE  GOODNESS  OP  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  2.0332  B=  .8618 

Z(K)=  .847  1.301  1.634  1.979 

LOGL=  -252.6468 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 

A  .1180  .0636  .0105  .0060  -.0018  -.0150 
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B 

.0636 

.0451 

.0029 

-  .0022 

- .0095 

-.0209 

z( 

1) 

.0105 

.0029 

.0095 

.0076 

.0065 

.0055 

z( 

2) 

.0060 

- .0022 

.0076 

.0127 

.0118 

.0117 

z( 

3) 

-.0018 

-  .0095 

.0065 

.0118 

.0183 

.0195 

z( 

4) 

-.0150 

-.0209 

.0055 

.0117 

.0195 

.0311 

CORRELATION  MATRIX: 

A 

1.0000 

.8724 

.3151 

.1545 

-.0378 

-.2466 

B 

.8724 

1.0000 

.1422 

-.0924 

-.3301 

-.5576 

Z{ 

1) 

.3151 

.1422 

1.0000 

.6951 

.4964 

.3203 

z{ 

2) 

.1545 

-.0924 

.6951 

1.0000 

.7784 

.5913 

z( 

3) 

-  .0378 

-.3301 

.4964 

.7784 

1.0000 

.8161 

z( 

4) 

-.2466 

-.5576 

.3203 

.5913 

.8161 

1.0000 

AREA  = 

.9382 

STD. 

DEV.  (AREA)  = 

.0174 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.4258 

( 

.2198  , 

.6549 

) 

.010 

.5111 

( 

.3173  , 

.7024 

) 

.020 

.6037 

( 

.4360  , 

.7539 

) 

.030 

.6598 

( 

.5117  , 

.7866 

) 

.040 

.6999 

( 

.5659  , 

.8112 

) 

.050 

.7308 

( 

.6073  , 

.8311 

) 

.060 

.7558 

( 

.6400  , 

.8479 

) 

.070 

.7767 

( 

.6667  , 

.8624 

) 

.080 

.7945 

( 

.6890  , 

.8752 

) 

.090 

.8099 

( 

.7078  , 

.8865 

) 

.100 

.8234 

( 

.7241  , 

.8965 

) 

.110 

.8355 

( 

.7383  , 

.9056 

) 

.120 

.8462 

( 

.7508  , 

.9137 

) 

.130 

.8560 

( 

.7619  , 

.9211 

) 

.140 

.8648 

( 

.7720  , 

.9277 

) 

.150 

.8729 

( 

.7810  , 

.9337 

) 

.200 

.9046 

( 

.8165  , 

.9567 

) 

.250 

.9268 

( 

.8419  , 

.9714 

) 

.300 

.9431 

( 

.8615  , 

.9811 

) 

.400 

.9653 

( 

.8909  , 

.9918 

) 

.500 

.9790 

( 

.9131  , 

.9966 

) 

.600 

.9878 

( 

.9311  , 

.9987 

) 

.700 

.9935 

( 

.9468  , 

.9996 

) 

.800 

.9971 

( 

.9612  , 

.9999 

) 

.900 

.9991 

( 

.9756  , 

1.0000 

) 

.950 

.9997 

( 

.9837  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 


CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0239, 

.6283) 

(  .0100, 

.5116) 

(  .0512, 

.7341) 

(  .0511, 

.7341) 

(  .0288, 

.6542) 

(  .0856, 

.8034) 

(  .0967, 

.8192) 

(  .0641, 

.7650) 

(  .1400, 

.8648) 

(  .1985, 

.9038) 

(  .1497, 

.8726) 

{  .2559, 

.9289) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


10:39:20 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  02  Digital  Calc  resp  for  296  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  216. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =80. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  162.  17.  3.  7.  27. 

ACTUALLY  POSITIVE  CASES  25.  1.  0,  2.  52. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .125  .157  .171  .250  1.000 

TPF:  .000  .650  .675  .675  .688  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.0483  B=  .5685 

Z(K)=  .674  .949  1.005  1.150 

LOGL=  -249.2581 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .6367  B=  .2172 

Z(K)=  .675  .940  .990  1.155 

LOGL=  -246.9307 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 

A  .0355  .0157  .0025  .0011  .0008  -.0002 
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B 

.0157 

.0165 

.0009 

-.0007 

-  .0011 

-.0024 

z( 

1) 

.0025 

.0009 

.0086 

.0074 

.0072 

.0066 

z( 

2) 

.0011 

-.0007 

.0074 

.0099 

.0097 

.0091 

z( 

3) 

.0008 

-  .0011 

.0072 

.0097 

.0103 

.0097 

z( 

4) 

-.0002 

- .0024 

.0066 

.0091 

.0097 

.0120 

CORRELATION  MATRIX: 

A 

1.0000 

.6478 

.1414 

.0599 

.0439 

-.0101 

B 

.6478 

1.0000 

.0750 

-.0577 

-.0841 

-  .1728 

z( 

1) 

.1414 

.0750 

1.0000 

.8018 

.7660 

.6527 

z( 

2) 

.0599 

-.0577 

.8018 

1.0000 

.9595 

.8322 

z( 

3) 

.0439 

-.0841 

.7660 

.9595 

1.0000 

.8697 

z( 

4) 

-.0101 

-.1728 

.6527 

.8322 

.8697 

1.0000 

AREA  =  .7331  STD.  DEV. (AREA)  =  .0571 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF 


.005 

.5308 

.010 

.5523 

.020 

.5756 

.030 

.5902 

.040 

.6012 

.050 

.6100 

.060 

.6175 

.070 

.6240 

.080 

.6299 

.090 

.6351 

.100 

.6400 

.110 

.6444 

.120 

.6486 

.130 

.6525 

.140 

.6562 

.150 

.6597 

.200 

.6751 

.250 

.6880 

.300 

.6995 

.400 

.7196 

.500 

.7378 

.600 

.7554 

.700 

.7735 

.800 

.7937 

.900 

.8199 

.950 

.8399 

(LOWER  BOUND, 

{  .3372  , 

{  .3762  , 

(  .4187  , 

(  .4450  , 

(  .4642  , 

(  .4792  , 

(  .4916  , 

(  .5020  , 

(  .5110  , 

(  .5189  , 

(  .5258  , 

{  .5321  , 

(  .5376  , 

{  .5427  , 

(  .5473  , 

(  .5515  , 

(  .5680  , 

(  .5795  , 

{  .5878  , 

(  .5988  , 

(  .6054  , 

(  .6096  , 

{  .6121  , 

(  .6132  , 

(  .6127  , 

(  .6111  , 


UPPER  BOUND) 

.7171  ) 
.7184  ) 
.7212  ) 
.7239  ) 
.7267  ) 
.7294  ) 
.7320  ) 
.7347  ) 
.7374  ) 
.7401  ) 
.7427  ) 
.7454  ) 
.7481  ) 
.7508  ) 
.7534  ) 
.7561  ) 
.7693  ) 
.7823  ) 
.7950  ) 
.8195  ) 
.8428  ) 
.8654  ) 
.8881  ) 
.9117  ) 
.9387  ) 
.9560  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 


CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

{  FPF  , 

TPF  ) 

(  .1241, 

.6502) 

(  .0855, 

.6328) 

(  .1736, 

.6673) 

(  .1611, 

.6634) 

(  .1172, 

.6475) 

{  .2146, 

.6790) 

(  .1737, 

.6674) 

(  .1282, 

.6518) 

(  .2284, 

.6826) 

(  .2498, 

.6879) 

(  .1957, 

.6739) 

{  .3108, 

.7018) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 


Appendix  HI  -  Reader 
Page  17 


Reader  02  Analog  ROC:  Type=FAD  N=400  Area=.5649  STD  Area=.0362 


False  Alarms 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


10:39:20 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OP  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  02  Analog  FAD  resp  for  296  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =270. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =26. 

RESPONSE  DATA: 

CATEGORY  1 

ACTUALLY  NEGATIVE  CASES  153. 

ACTUALLY  POSITIVE  CASES  4. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .119  .200  .256  .433  1.000 

TPF:  .000  .423  .615  .769  .846  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3239  B=  1.2078 

Z(K)=  .168  .657  .841  1.183 

LOGL=  -375.9875 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.3336  B=  1.2586 

Z(K)=  .171  .637  .842  1.191 

LOGL=  -375.7867 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

VARIANCE -COVARIANCE  MATRIX: 

A  .1254  .0888  .0078  .0065  .0052  .0018 

B  .0888  .1135  .0032  -.0003  -.0028  -.0087 


2 

48. 

2. 


3 

15. 

4. 


4 

22. 

5. 


5 

32. 

11. 
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z( 

1) 

.0078 

.0032 

.0059 

.0043 

.0038 

.0032 

z( 

2) 

.0065 

-.0003 

.0043 

.0066 

.0059 

.0051 

z( 

3) 

.0052 

-.0028 

.0038 

.0059 

.0073 

.0064 

z( 

4) 

.0018 

-  .0087 

.0032 

.0051 

.0064 

.0098 

CORRELATION  MATRIX: 

A 

1.0000 

.7447 

.2857 

.2258 

.1704 

.0501 

B 

.7447 

1.0000 

.1252 

-.0093 

-.0969 

-.2591 

z( 

1) 

.2857 

.1252 

1.0000 

.6971 

.5856 

.4163 

z( 

2) 

.2258 

-.0093 

.6971 

1.0000 

.8519 

.6345 

z( 

3) 

.1704 

-.0969 

.5856 

.8519 

1.0000 

.7562 

z( 

4) 

.0501 

-.2591 

.4163 

.6345 

.7562 

1.0000 

AREA  =  .7966  STD.  DEV.  (AREA)  =  .0423 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.0281 

( 

.0007  , 

.2619 

) 

.010 

.0554 

( 

.0033  , 

.3173 

) 

.020 

.1053 

( 

.0135  , 

.3847 

) 

.030 

.1505 

( 

.0291  , 

.4311 

) 

.040 

.1921 

( 

.0485  , 

.4678 

) 

.050 

.2305 

( 

.0706  , 

.4988 

) 

.060 

.2664 

( 

.0947  , 

.5260 

) 

.070 

.3001 

( 

.1199  , 

.5505 

) 

.080 

.3317 

( 

.1459  , 

.5730 

) 

.090 

.3616 

( 

.1721  , 

.5940 

) 

.100 

.3899 

( 

.1982  , 

.6136 

) 

.110 

.4167 

( 

.2241  , 

.6323 

) 

.120 

.4422 

( 

.2495  , 

.6500 

) 

.130 

.4665 

( 

.2743  , 

.6670 

) 

.140 

.4896 

( 

.2983  , 

.6833 

) 

.150 

.5116 

( 

.3215  , 

.6991 

) 

.200 

.6082 

( 

.4245  , 

.7702 

) 

.250 

.6862 

( 

.5065  , 

.8299 

) 

.300 

.7499 

( 

.5715  , 

.8786 

) 

.400 

.8450 

( 

.6681  , 

.9447 

) 

.500 

.9088 

( 

.7388  , 

.9787 

) 

.600 

.9507 

( 

.7956  , 

.9934 

) 

.700 

.9769 

( 

.8447  , 

.9985 

) 

.800 

.9916 

( 

.8896  , 

.9998 

) 

.900 

.9984 

( 

.9341  , 

1.0000 

) 

.950 

.9997 

( 

.9587  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 


CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 


( 

FPF  , 

TPF  ) 

( 

( 

.1169, 

.4346) 

( 

( 

.2000, 

.6081) 

( 

( 

.2621, 

.7026) 

( 

( 

.4323, 

.8684) 

( 

LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

.0831, 

.3412) 

(  .1596, 

.5318) 

.1564, 

.5251) 

(  .2502, 

.6864) 

.2131, 

.6300) 

(  .3163, 

.7679) 

.3742, 

.8238) 

(  .4919, 

.9046) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader02  Digital  ROC:  Type=FAD  Area=.4210  STD  Area=.0487 


False  Alarms 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 


next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the 
(Y/N,  or  R  to  restart) 

11/  3/1998 


ROC  curve? 


DATE  - 


TIME  - 


10:39:20 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  02  Digital  FAD  resp  for  296  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  270. 
NO.  OF  ACTUALLY  POSITIVE  CASES  =  26. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  135.  44. 

CASES  8 .  7 . 


3  4  5 

35.  23.  33. 

4.  2.  5. 


OBSERVED  OPERATING  POINTS: 

FPF:  .000  .122  .207  .337  .500  1.000 

TPF:  .000  .192  .269  .423  .692  1.000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  .4108  B=  1.1749 

Z(K)=  .000  .420  .815  1.164 

LOGL=  -411.1221 

CHI-SQUARE  GOODNESS  OP  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .4477  B=  1.2104 

Z(K)=  -.003  .434  .823  1.154 

LOGL=  -411.0256 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 

A  .0710  .0409  .0069  .0057  .0043  .0030 
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B 

.0409 

.0857 

.0032 

-.0001 

-.0036 

-.0067 

z{ 

1) 

.0069 

.0032 

.0058 

.0043 

.0034 

.0028 

z{ 

2) 

.0057 

-  .0001 

.0043 

.0061 

.0049 

.0042 

z( 

3) 

.0043 

- .0036 

.0034 

.0049 

.0073 

.0064 

z( 

4) 

.0030 

-  .0067 

.0028 

.0042 

.0064 

.0095 

CORRELATION  MATRIX: 

A 

1.0000 

.5250 

.3392 

.2762 

.1905 

.1150 

B 

.5250 

1.0000 

.1452 

- .0043 

- .1442 

-.2359 

Z( 

1) 

.3392 

.1452 

1.0000 

.7201 

.5163 

.3744 

z( 

2) 

.2762 

-.0043 

.7201 

1.0000 

.7378 

.5567 

z( 

3) 

.1905 

-.1442 

.5163 

.7378 

1.0000 

.7657 

z( 

4) 

.1150 

-.2359 

,3744 

.5567 

.7657 

1.0000 

AREA  =  .6122  STD.  DEV. (AREA)  =  .0583 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.0038 

( 

.010 

.0089 

( 

.020 

.0207 

( 

.030 

.0337 

( 

.040 

.0473 

( 

.050 

.0613 

( 

.060 

.0757 

( 

.070 

.0903 

{ 

.080 

.1050 

( 

.090 

.1199 

( 

.100 

.1348 

( 

.110 

.1498 

{ 

.120 

.1649 

{ 

.130 

.1799 

( 

.140 

.1949 

( 

.150 

.2099 

{ 

.200 

.2841 

( 

.250 

.3563 

( 

.300 

.4260 

( 

.400 

.5563 

( 

.500 

.6728 

{ 

.600 

.7745 

( 

.700 

.8604 

( 

.800 

.9287 

( 

.900 

.9772 

( 

.950 

.9926 

( 

.0000  , 

.0826 

) 

.0002  , 

.1115 

) 

.0012  , 

.1512 

) 

.0030  , 

.1814 

) 

.0058  , 

.2070 

) 

.0095  , 

.2296 

) 

.0141  , 

.2503 

) 

.0195  , 

.2695 

) 

.0258  , 

.2877 

) 

.0328  , 

.3050 

) 

.0406  , 

.3216 

) 

.0489  , 

.3376 

) 

.0579  , 

.3532 

) 

.0674  , 

.3684 

) 

.0774  , 

.3832 

) 

.0878  , 

.3978 

) 

.1442  , 

.4682 

) 

.2042  , 

.5359 

) 

.2638  , 

.6020 

) 

.3740  , 

.7272 

) 

.4703  , 

.8339 

) 

.5568  / 

.9138 

) 

.6389  , 

.9647 

) 

.7219  , 

.9905 

) 

.8149  , 

.9990 

) 

.8734  , 

.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 


CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

{  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .1243, 

.1714) 

{  .0894, 

.1190) 

{  .1679, 

.2365) 

{  .2052, 

.2917) 

(  .1610, 

.2263) 

(  .2560, 

.3647) 

(  .3322, 

.4692) 

(  .2788, 

.3967) 

(  .3893, 

.5427) 

(  .5013, 

.6742) 

(  .4419, 

.6067) 

(  .5607, 

.7365) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


10:39:20 


R  0  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  02  Analog  Diag  resp  for  296  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  215. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  81. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .000  .056  .516  .884  1.000 

TPF:  .000  .123  .432  .938  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.4189  B=  .9291 

Z(K)=  -1.194  -.041  1.591  2.831 
LOGL=  -338.0848 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.4837  B=  .9945 

Z(K)=  -1.203  -.040  1.625  2.707 
LOGL=  -337.7247 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1 

25. 

0. 


2 

79. 

5. 


3 

99. 

41. 


4 

12. 

25. 


5 

0. 

10. 


VARIANCE -COVARIANCE  MATRIX: 

A  .0452  .0185  .0042  .0063  .0057  -.0075 
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B 

.0185 

.0184 

.0018 

.0019 

-  .0070 

-.0251 

Z( 

1) 

.0042 

.0018 

.0125 

.0037 

.0013 

-.0003 

z( 

2) 

.0063 

.0019 

.0037 

.0072 

.0030 

.0012 

z( 

3) 

.0057 

-.0070 

.0013 

.0030 

.0187 

.0235 

z( 

4) 

- .0075 

-.0251 

- .0003 

.0012 

.0235 

.0726 

CORRELATION  MATRIX; 

A 

1.0000 

.6406 

.1766 

.3476 

.1971 

-.1309 

B 

,6406 

1.0000 

.1160 

.1637 

-.3791 

-.6859 

z{ 

1) 

.1766 

.1160 

1.0000 

.3877 

.0855 

-.0113 

z( 

2) 

.3476 

.1637 

.3877 

1.0000 

.2611 

.0542 

z( 

3) 

.1971 

-  .3791 

.0855 

.2611 

1.0000 

.6371 

z( 

4) 

-.1309 

-  .6859 

-.0113 

.0542 

.6371 

1.0000 

AREA  =  .8536  STD.  DEV.  (AREA)  =  .0272 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION; 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.1405 

{ 

.0543  , 

.2906 

) 

.010 

.2032 

( 

.0958  , 

.3615 

) 

.020 

.2880 

( 

.1626  , 

.4466 

) 

.030 

.3493 

( 

.2167  , 

.5036 

) 

.040 

.3983 

( 

.2629  , 

.5474 

) 

.050 

.4394 

( 

.3032  , 

.5833 

) 

.060 

.4750 

( 

.3391  , 

.6138 

) 

.070 

.5063 

( 

.3714  , 

.6404 

) 

.080 

.5343 

( 

.4008  , 

.6641 

) 

,090 

.5597 

{ 

.4277  , 

.6853 

) 

.100 

.5828 

( 

.4525  , 

.7046 

) 

.110 

.6040 

( 

.4754  , 

.7222 

) 

.120 

.6237 

( 

.4967  , 

.7384 

) 

.130 

.6419 

( 

.5166  , 

.7534 

) 

.140 

.6588 

( 

.5353  , 

.7673 

) 

.150 

.6747 

( 

.5528  , 

.7803 

) 

.200 

.7412 

( 

.6270  , 

.8340 

) 

.250 

.7920 

( 

.6849  , 

.8739 

) 

.300 

.8321 

( 

.7317  , 

.9044 

) 

.400 

.8911 

( 

.8037  , 

.9462 

) 

.500 

.9311 

( 

.8570  , 

.9713 

) 

.600 

.9587 

( 

.8984  , 

.9860 

) 

.700 

.9775 

( 

.9315  , 

.9942 

) 

.800 

.9898 

( 

.9586  , 

.9982 

) 

.900 

.9971 

( 

.9808  , 

.9997 

) 

.950 

.9991 

( 

.9905  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 


CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS; 


EXPECTED  OPERATING  POINT 


( 

FPF  , 

TPF  ) 

( 

( 

.0034, 

.1135) 

( 

( 

.0521, 

.4473) 

( 

( 

.5159, 

.9361) 

( 

( 

.8855, 

.9963) 

( 

LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

0006, 

.0415) 

(  .0147, 

.2474) 

0292, 

.3449) 

(  .0874, 

.5533) 

4497, 

.9128) 

{  .5816, 

.9543) 

8375, 

.9931) 

(  .9225, 

.9981) 

Do  you  want  to  use.  data  from  a  previously  created  input  file  for  the 
next  run? 

(y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader02  Digital  ROC:  Type=Diag  Area=.6949  STD  Area=.0398 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


10:39:20 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  02  Digital  Diag  resp  for  296  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  215. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  81. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .019  .107  .535  .912  1.000 

TPF:  .000  .173  .494  .877  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.1416  B=  .9873 

Z(K)=  -1.351  -.087  1.243  2.084 

LOGL=  -371.1523 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.1635  B=  .9946 

Z(K)=  -1.365  -.072  1.217  2.110 

LOGL=  -370.9608 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1 

19. 

0. 


2 

81. 

10. 


3 

92. 

31. 


4 

19. 

26. 


5 

4. 

14. 


VARIANCE -COVARIANCE  MATRIX: 

A  .0323  .0127  .0043  ,0060  .0055  .0008 
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B 

.0127 

.0158 

.0022 

.0018 

- .0035 

-.0130 

z( 

1) 

.0043 

.0022 

.0145 

.0036 

.0015 

.0001 

z{ 

2) 

.0060 

.0018 

.0036 

.0071 

.0035 

.0021 

z( 

3) 

.0055 

-  .0035 

.0015 

.0035 

.0116 

.0118 

Z( 

4) 

.0008 

-.0130 

.0001 

.0021 

.0118 

.0327 

CORRELATION  MATRIX: 

A 

1.0000 

.5613 

.1988 

.3937 

.2843 

.0257 

B 

.5613 

1.0000 

.1433 

.1724 

-  .2617 

--*5733 

z( 

1) 

.1988 

.1433 

1.0000 

.3532 

.1166 

.0067 

Z( 

2) 

.3937 

.1724 

.3532 

1.0000 

.3836 

.1393 

z( 

3) 

.2843 

-  .2617 

.1166 

.3836 

1.0000 

.6077 

z{ 

4) 

.0257 

-.5733 

.0067 

.1393 

.6077 

1.0000 

AREA  =  .7953  STD.  DEV.  (AREA)  =  .0305 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF 


005 

.0809 

010 

.1249 

020 

.1895 

030 

.2396 

040 

.2816 

050 

.3182 

060 

.3508 

070 

.3803 

080 

.4074 

090 

.4324 

100 

.4557 

110 

.4774 

120 

.4979 

130 

.5172 

140 

.5354 

150 

.5528 

200 

.6280 

250 

.6890 

300 

.7397 

400 

.8191 

500 

.8777 

600 

.9215 

700 

.9540 

800 

.9773 

900 

.9926 

950 

.9974 

(LOWER  BOUND, 

(  .0272 

(  .0520 

(  .0962 

(  .1353 

(  .1707 

(  .2031 

(  .2332 

(  .2611 

(  .2873 

(  .3119 

(  .3351 

(  .3571 

(  .3780 

(  .3978 

(  .4167 

(  .4347 

(  .5139 

(  .5791 

(  .6341 

(  .7225 

(  .7914 

(  .8472 

(  .8937 

(  .9334 

(  .9677 

(  .9834  , 


UPPER  BOUND) 

.1910  ) 

.2496  ) 

.3243  ) 

.3770  ) 

.4189  ) 

.4541  ) 

.4849  ) 

.5122  ) 

.5369  ) 

.5596  ) 

.  .5805  ) 

.5999  ) 

.6180  ) 
.6351  ) 

.6511  ) 

.6663  ) 

.7318  ) 

.7841  ) 

.8269  ) 

.8913  ) 

.9352  ) 

.9645  ) 

.9831  ) 

.9938  ) 

.9988  ) 

.9997  ) 


ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 


CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0174, 

.1748) 

(  .0069, 

.0989) 

(  .0396, 

.2800) 

(  .1118, 

.4813) 

(  .0767, 

.3987) 

(  .1573, 

.5648) 

(  .5288, 

.8917) 

{  .4629, 

.8579) 

(  .5940, 

.9192) 

(  .9139, 

.9942) 

{  .8705, 

.9889) 

(  .9453, 

.9971) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader  03  analog  ROC  Type=mass  N=329  Area=.8717  STD  Area=.0419 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 
12:21:  6 

R  O  C  F  I  T  (IBM  VERSION  1.2) 


TIME  - 


MAXIMUM  LIKELIHOOD  ES 
OF  A  BINORMAL  ROC 

FROM  RATING  DAT 


TIMATION 

CURVE 

A 


DATA  DESCRIPTION:  Reader  03  Analog  Mass  resp  for  329  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =280. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  49. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  175.  71.  5.  20.  9. 

ACTUALLY  POSITIVE  CASES  7.  3.  1.  5.  33. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .032  .104  .121  .375  1.000 

TPF:  .000  .673  .776  .796  .857  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2298  B=  .3955 

Z(K)=  .318  1.168  1.262  1.851 

LOGL=  -334.4240 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.2248  B=  .4116 

Z(K)=  .320  1.154  1.255  1.869 

LOGL=  -334.3403 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0627 

.0224 

.0026 

.0017 

.0015 

-.0014 

B 

.0224 

.0169 

.0008 

-.0007 

-.0011 

-.0054 

Z( 

1) 

.0026 

.0008 

.0058 

.0036 

.0034 

.0026 

z( 

2) 

.0017 

-  .0007 

.0036 

.0090 

.0086 

.0071 

z{ 

3) 

.0015 

-  .0011 

.0034 

.0086 

.0099 

.0082 

Z( 

4) 

-.0014 

-.0054 

.0026 

.0071 

.0082 

.0214 

CORRELATION  MATRIX: 

A  1.0000  .6880  .1362  .0713  .0592  -.0388 
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B 

.6880 

1.0000 

.0845 

-  .0581 

- .0830 

-.2826 

z( 

1) 

.1362 

,0845 

1.0000 

.4970 

.4519 

.2297 

z( 

2) 

.0713 

-.0581 

.4970 

1.0000 

.9149 

.5114 

z( 

3) 

.0592 

- .0830 

.4519 

.9149 

1.0000 

.5627 

z( 

4) 

-.0388 

-.2826 

.2297 

.5114 

.5627 

1.0000 

AREA  =  .8713  STD.  DEV.  (AREA)  =  .0419 

ESTIMATED  BINORMAIi  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.5653 

( 

.3771  , 

.7396 

) 

.010 

.6053 

( 

.4322  , 

.7596 

) 

.020 

.6478 

( 

.4912  , 

.7825 

) 

.030 

.6738 

( 

.5269  , 

.7978 

) 

.040 

.6929 

( 

.5525  , 

.8096 

) 

.050 

.7080 

( 

.5723  , 

.8194 

) 

.060 

.7206 

( 

.5885  , 

.8279 

) 

.070 

.7315 

( 

.6022  , 

.8354 

) 

.080 

.7410 

( 

.6139  , 

.8421 

) 

.090 

.7495 

( 

.6242  , 

.8483 

) 

.100 

.7572 

( 

.6334  , 

.8540 

) 

.110 

.7642 

( 

.6416  , 

.8593 

) 

.120 

.7707 

( 

.6491  , 

.8642 

) 

.130 

.7767 

( 

.6559  , 

.8689 

) 

.140 

.7824 

( 

.6622  , 

.8733 

) 

.150 

.7876 

( 

.6680  , 

.8774 

) 

.200 

.8102 

( 

.6920  , 

.8954 

) 

.250 

.8283 

( 

.7103  , 

.9100 

) 

.300 

.8436 

( 

.7252  , 

.9222 

) 

.400 

.8688 

( 

.7491  , 

.9418 

) 

.500 

.8897 

( 

.7685  , 

.9569 

) 

.600 

.9081 

( 

.7857  , 

.9690 

) 

.700 

.9251 

( 

.8023  , 

.9789 

) 

.800 

.9419 

( 

.8197  , 

.9871 

) 

.900 

.9601 

( 

.8411  , 

.9939 

) 

.950 

.9714 

( 

.8567  , 

.9969 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .0308, 

.6757) 

(  .0156, 

.6322) 

( 

.0568, 

.7169) 

(  .1048, 

.7606) 

(  .0736, 

.7350) 

( 

.1446, 

.7848) 

(  .1243, 

.7733) 

(  .0901, 

.7496) 

( 

.1666, 

.7957) 

(  .3744, 

.8628) 

(  .3192, 

.8488) 

( 

.4322, 

.8759) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  qpiit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/n,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


12:21:  7 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  03  Analog  Calc  resp  for  329  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  253. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  76. 


RESPONSE  DATA: 


CATEGORY 

1 

2 

3 

4 

5 

ACTUALLY 

NEGATIVE 

CASES 

215. 

21. 

4. 

3. 

10 

ACTUALLY 

POSITIVE 

CASES 

23. 

0. 

1. 

5. 

47 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .040  .051  .067  .150  1.000 

TPF:  .000  .618  .684  .697  .697  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  1.0242  B=  .3688 

Z{K)=  1.036  1.497  1.632  1.757 

LOGL=  -225.3118 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .8249  B=  .2857 

Z(K)=  1.039  1.434  1.566  1.813 

LOGL=  -223.2512 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0521 

.0214 

.0037 

.0007 

-.0007 

-.0042 

B 

.0214 

.0148 

.0010 

-.0015 

-.0027 

-.0058 

z( 

1) 

.0037 

.0010 

.0093 

.0077 

.0073 

.0065 

z( 

2) 

.0007 

-  .0015 

.0077 

.0131 

.0126 

.0120 

z{ 

3) 

-.0007 

-.0027 

.0073 

.0126 

.0153 

.0148 

z( 

4) 

-.0042 

-.0058 

.0065 

.0120 

.0148 

.0220 

CORRELATION  MATRIX: 

A  1.0000  .7704  .1685  .0266  -.0249  -.1240 
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B 

.7704 

1.0000 

.0869 

-  .1098 

-  .1816 

-.3196 

z{ 

1) 

.1685 

.0869 

1.0000 

.6994 

.6089 

.4564 

z{ 

2) 

.0266 

-.1098 

.6994 

1.0000 

.8892 

.7062 

z{ 

3) 

-.0249 

-  .1816 

.6089 

.8892 

1.0000 

.8039 

z( 

4) 

-.1240 

-.3196 

.4564 

.7062 

.8039 

1.0000 

AREA  = 

.7862 

STD. 

DEV.  (AREA)  = 

.0584 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.5354 

( 

.3808  , 

.6848 

) 

.010 

.5636 

( 

.4231  , 

.6964 

) 

.020 

.5941 

( 

.4674  , 

.7115 

) 

.030 

.6131 

( 

.4936  , 

.7227 

) 

.040 

.6273 

( 

.5121  , 

.7320 

) 

.050 

.6386 

( 

.5262  , 

.7402 

) 

.060 

.6482 

( 

.5375  , 

.7476 

) 

.070 

.6566 

( 

.5469  , 

.7544 

) 

.080 

.6640 

( 

.5549  , 

.7608 

) 

.090 

.6707 

( 

.5618  , 

.7667 

) 

.100 

.6768 

( 

.5678  , 

.7724 

) 

.110 

.6824 

( 

.5732  , 

.7777 

) 

.120 

.6876 

( 

.5780  , 

.7828 

) 

.130 

.6925 

( 

.5823  , 

.7877 

) 

.140 

.6971 

( 

.5863  , 

.7923 

) 

.150 

.7015 

( 

.5899  , 

.7968 

) 

.200 

.7205 

( 

.6044  , 

.8170 

) 

.250 

.7364 

( 

.6152  , 

.8344 

) 

.300 

.7502 

( 

.6236  , 

.8497 

) 

.400 

.7742 

( 

.6368  , 

.8760 

) 

.500 

.7953 

( 

.6471  , 

.8984 

) 

-600 

.8152 

( 

.6561  , 

.9181 

) 

.700 

.8351 

( 

.6646  , 

.9362 

) 

.800 

.8566 

( 

.6736  , 

.9536 

) 

.900 

.8832 

( 

.6848  , 

.9714 

) 

.950 

.9023 

( 

.6932  , 

.9815 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .0349, 

.6206) 

(  .0177, 

.5886) 

( 

.0640, 

.6518) 

(  .0586, 

.6470) 

(  .0352, 

.6210) 

( 

.0928, 

.6725) 

(  .0758, 

.6610) 

(  .0486, 

.6373) 

( 

.1132, 

.6841) 

(  .1493, 

.7012) 

(  .1097, 

.6822) 

( 

.1976, 

.7197) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 


(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


12:21:  7 


R  0  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  03  Analog  FAD  resp  for  329  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
{E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  307. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =22. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  190.  46.  1.  42.  28. 

ACTUALLY  POSITIVE  CASES  7 .  2.  2.  7 .  4 . 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .091  .228  .231  .381  1.000 

TPF:  .000  .182  .500  .591  .682  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.0184  B=  1.3739 

Z(K)=  .302  .734  .745  1.334 

LOGL=  -372.1085 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .9435  B=  1.3392 

Z(K)=  .305  .712  .741  1.341 

LOGL=  -370.9217 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1309 

.0930 

.0079 

.0059 

.0057 

.0007 

B 

.0930 

.1248 

.0031 

-.0007 

-.0010 

-  .0091 

z( 

1) 

.0079 

.0031 

.0053 

.0041 

.0040 

.0029 

z( 

2) 

.0059 

-.0007 

.0041 

.0060 

.0059 

.0046 

z( 

3) 

.0057 

-.0010 

.0040 

.0059 

.0061 

.0047 

z( 

4) 

.0007 

-.0091 

.0029 

.0046 

.0047 

.0100 

CORREIiATION  MATRIX; 

A 

1.0000 

.7279 

.3011 

.2114 

.2032 

.0188 

B 

,7279 

1.0000 

.1200 

-  .0257 

-  .0373 

-  .2570 
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1) 

.3011 

.1200 

1.0000 

.7277 

.7105 

.4008 

2) 

,2114 

-  .0257 

.7277 

1.0000 

.9779 

.5860 

3) 

.2032 

-.0373 

.7105 

.9779 

1.0000 

,6010 

4) 

.0188 

-  .2570 

.4008 

.5860 

.6010 

1.0000 

AREA  = 

.7138 

STD. 

DEV.  (AREA)  = 

.0547 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.0061 

( 

.0001  , 

.1253 

) 

.010 

.0149 

( 

.0004  , 

.1649 

) 

.020 

.0353 

( 

.0023  , 

.2179 

) 

.030 

.0575 

( 

,0062  , 

.2574 

) 

.040 

.0805 

( 

.0122  , 

.2904 

) 

.050 

.1039 

( 

.0202  , 

.3193 

) 

.060 

.1273 

( 

.0300  , 

.3457 

) 

.070 

.1507 

( 

.0414  , 

.3701 

) 

.080 

.1740 

( 

.0542  , 

.3931 

) 

.090 

.1970 

( 

,0681  , 

.4149 

) 

.100 

.2197 

( 

.0830  , 

.4360 

) 

.110 

.2422 

( 

.0987  , 

.4563 

) 

.120 

.2643 

( 

.1150  , 

.4760 

) 

.130 

.2860 

( 

.1317  , 

.4953 

) 

.140 

.3074 

( 

.1487  , 

.5141 

) 

.150 

.3283 

( 

.1658  , 

.5326 

) 

.200 

.4272 

( 

.2501  , 

.6207 

) 

.250 

.5162 

( 

.3270  , 

.7017 

) 

.300 

.5955 

( 

.3943  , 

.7739 

) 

.400 

.7273 

( 

.5043  , 

.8847 

) 

.500 

.8273 

( 

.5926  , 

.9508 

) 

.600 

.9001 

( 

.6689  , 

.9833 

) 

.700 

.9500 

( 

.7391  , 

.9960 

) 

.800 

.9808 

( 

.8075  , 

.9995 

) 

.900 

.9961 

( 

.8799  , 

1.0000 

) 

.950 

.9992 

( 

.9224  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0900, 

.1970) 

(  .0621, 

.1323) ■ 

(  .1262, 

.2778) 

(  .2294, 

.4805) 

(  .1857, 

.3999) 

(  .2783, 

.5620) 

(  .2381, 

.4958) 

(  .1937, 

.4153) 

(  .2876, 

.5765) 

(  .3803, 

.7038) 

(  .3273, 

.6347) 

(  .4356, 

.7662) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 


Appendix  HI  -  Reader  3 
Page  10 


Reader  03  analog  ROC  Type^diag  N=329  Area=.8105  STD  Area=.0307 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  rxm? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/N,  or  R  to  restart) 

DATE  - 


11/  3/1998 


TIME  - 


12:21;  7 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  03  Analog  Diag  resp  for  329  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  240. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  89. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF;  .000  .004  .058  .171  .479  1.000 

TPF:  .000  .202  .449  .663  .854  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.0887  B=  .7372 

Z(K)=  .052  .951  1.569  2.639 

LOGL=  -413.4673 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.1018  B=  .7541 

Z(K)=  .053  .940  1.599  2.573 

LOGL=  -413.2282 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

125.  74.  27.  13.  1. 

13.  17.  19.  22.  18. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0295 

.0120 

.0047 

.0040 

.0018 

-.0067 

B 

.0120 

.0120 

.0016 

-.0008 

- .0050 

-.0178 

z( 

1) 

.0047 

.0016 

.0065 

.0038 

.0027 

.0010 

z{ 

2) 

.0040 

-.0008 

.0038 

.0083 

.0070 

.0073 

z( 

3) 

.0018 

- .0050 

.0027 

.0070 

.0154 

.0184 

z( 

4) 

-  .0067 

- .0178 

.0010 

.0073 

.0184 

.0591 

CORRELATION  MATRIX: 

A  1.0000  .6383  .3413  .2531  .0846  -.1610 
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B 

,6383 

1.0000 

.1823 

- .0772 

-  .3656 

-.6665 

z( 

1) 

.3413 

.1823 

1.0000 

.5214 

.2702 

.0492 

z( 

2) 

.2531 

-.0772 

.5214 

1.0000 

.6220 

.3268 

z( 

3) 

.0846 

-.3656 

.2702 

.6220 

1.0000 

.6120 

z( 

4) 

-.1610 

-.6665 

.0492 

.3268 

.6120 

1.0000 

AREA  = 

.8105 

STD. 

DEV.  (AREA)  = 

.0307 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION; 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.2002 

( 

.1024  , 

.3394 

) 

.010 

.2569 

( 

.1495  , 

.3947 

) 

.020 

.3273 

( 

.2142  , 

.4591 

) 

.030 

.3757 

( 

.2617  , 

.5018 

) 

.040 

.4134 

( 

.3000  , 

.5346 

) 

.050 

.4448 

( 

.3325  , 

.5617 

) 

.060 

.4717 

( 

.3607  , 

.5850 

) 

.070 

.4955 

( 

.3858  , 

.6055 

) 

.080 

.5168 

( 

.4084  , 

.6239 

) 

.090 

.5361 

( 

.4290  , 

.6406 

) 

.100 

.5538 

( 

.4478  , 

.6560 

) 

.110 

.5701 

( 

.4653  , 

.6703 

) 

.120 

.5854 

( 

.4815  , 

.6836 

) 

.130 

.5996 

( 

.4966  , 

.6961 

) 

.140 

.6130 

( 

,5108  , 

.7078 

) 

.150 

.6256 

( 

.5242  , 

.7189 

) 

.200 

.6798 

( 

.5815  , 

.7669 

) 

.250 

.7235 

( 

.6273  , 

.8057 

) 

.300 

.7601 

( 

.6656  , 

.8378 

) 

.400 

.8189 

( 

.7277  , 

.8880 

) 

.500 

.8647 

( 

.7778  , 

.9249 

) 

.600 

.9019 

( 

.8208  , 

.9522 

) 

.700 

.9328 

( 

.8597  , 

.9722 

) 

.800 

.9587 

( 

.8967  , 

.9864 

) 

.900 

.9807 

( 

.9349  , 

.9956 

) 

.950 

.9904 

( 

.9570  , 

.9985 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 


(  FPF  ,  TPF  ) .  ( 

(  .0050,  .2008)  ( 

(  .0549,  .4585)  ( 

(  .1736,  .6529)  ( 

(  .4787,  .8558)  ( 


LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

0011, 

.1154) 

(  .0180, 

.3159) 

0327, 

.3869) 

(  .0875, 

.5315) 

1316, 

.6018) 

(  .2234, 

.7013) 

4162, 

.8270) 

(  .5418, 

.8812) 

Do  you  want  to  use  data  from  a  previously  created  input 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader04  Analog  ROC  Type=Mass  N=345  Area=.8809  STD  Area=.0290 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/N,  or  R  to  restart) 

DATE  - 


8/16/1998 
9:  7 

R  O  C  F  I  T  (IBM  VERSION  1.2) 


TIME  -  13: 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION;  Reader  04  Analog  Mass  for  345  cases  currently 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  285. 

NO.  OF  ACTUiUiLY  POSITIVE  CASES  =  60. 

RESPONSE  DATA: 

CATEGORY  123 

ACTUALLY  NEGATIVE  CASES  51.  113.  90. 

ACTUALLY  POSITIVE  CASES  1.  6.  9. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .039  .109  .425  .821  1.000 

TPF:  .000  .617  .733  .883  .983  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.4413  B=  .6718 

Z(K)=  -.919  .190  1.233  1.768 

LOGL=  -453.2962 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.3920  B=  .6270 

Z(K)=  -.924  .197  1.228  1.758 

LOGL=  -453.1865 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0445 

.0165 

.0024 

.0027 

.0026 

.0011 

B 

.0165 

.0150 

.0011 

.0006 

-.0015 

-  .0052 

z{ 

1) 

.0024 

.0011 

.0075 

.0026 

.0014 

.0010 

z{ 

2) 

.0027 

.0006 

.0026 

.0055 

.0031 

.0025 

Z( 

3) 

.0026 

-.0015 

.0014 

.0031 

.0094 

.0082 

z{ 

4) 

.0011 

-.0052 

.0010 

.0025 

.0082 

.0172 

CORRELATION  MATRIX: 

A 

1.0000 

.6391 

.1335 

.1694 

.1284 

.0391 

B 

.6391 

1.0000 

.1038 

.0628 

-.1301 

-  .3247 

4  5 

20.  11. 

7.  37. 
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z( 

1) 

.1335 

.1038 

1.0000 

.4097 

.1681 

.0837 

z( 

2) 

.1694 

.0628 

.4097 

1.0000 

.4306 

.2563 

z( 

3) 

.1284 

-  .1301 

.1681 

.4306 

1.0000 

.6454 

z( 

4) 

.0391 

-  .3247 

.0837 

.2563 

.6454 

1.0000 

AREA  =  .8809  STD.  DEV. (AREA)  =  .0290 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.4116 

( 

.2422  , 

.5997 

) 

.010 

.4733 

( 

.3084  , 

.6429 

) 

.020 

.5414 

( 

.3867  , 

.6900 

) 

.030 

.5841 

( 

.4378  , 

.7196 

) 

.040 

.6156 

( 

.4760  , 

.7416 

) 

.050 

.6407 

( 

.5066  , 

.7593 

) 

.060 

.6616 

( 

.5322  , 

.7743 

) 

.070 

.6796 

( 

.5541  , 

.7873 

) 

.080 

.6953 

( 

.5732  , 

.7987 

) 

.090 

.7093 

( 

.5902  , 

.8090 

) 

.100 

.7219 

( 

.6054  , 

.8184 

) 

.110 

.7333 

( 

.6193  , 

.8270 

) 

.120 

.7438 

( 

.6319  , 

.8349 

) 

.130 

.7536 

( 

.6435  , 

.8422 

) 

.140 

.7626 

( 

.6542  , 

.8491 

) 

.150 

.7710 

( 

.6642  , 

.8555 

) 

.200 

.8063 

( 

.7056  , 

.8827 

) 

.250 

.8338 

( 

.7374  , 

.9038 

) 

.300 

.8562 

( 

.7632  , 

.9208 

) 

.400 

.8913 

( 

.8040  , 

.9464 

) 

.500 

.9180 

( 

.8361  , 

.9645 

) 

.600 

.9395 

( 

.8635  , 

.9775 

) 

.700 

.9573 

( 

.8883  , 

.9869 

) 

.800 

.9726 

( 

.9124  , 

.9935 

) 

.900 

.9859 

( 

.9384  , 

.9978 

) 

.950 

.9923 

( 

.9547  , 

.9992 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .0393, 

.6139) 

(  .0219, 

.5510) 

( 

.0667, 

.6739) 

(  .1098, 

.7331) 

(  .0782, 

.6926) 

( 

.1496, 

.7707) 

(  .4218, 

.8976) 

(  .3658, 

.8804) 

( 

.4793, 

.9130) 

(  .8223, 

.  9757) 

(  .7746, 

.9689) 

( 

.8631, 

.9812) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/N,  or  R  to  restart) 

DATE  - 


8/16/1998 


TIME  - 


13:13:21 


R  0  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  04  Digital  Mass  resp  for  the  300  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  285. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =60. 

RESPONSE  DATA: 

CATEGORY  1  2  3  4  5 

ACTUALLY  NEGATIVE  CASES  20.  101.  97.  42.  25. 

ACTUALLY  POSITIVE  CASES  0.  4.  12.  7.  37. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .088  .235  .575  .930  1.000 

TPF:  .000  .617  .733  .933  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2825  B=  .7648 

Z(K)=  -1.475  -.190  .722  1.355 

LOGL=  -468.1990 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.3224  B=  .7901 

Z{K)=  -1.484  -.190  .742  1.330 

LOGL=  -468.0175 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0418 

.0185 

.0027 

.0033 

.0034 

.0023 

B 

.0185 

.0225 

.0016 

.0012 

-  .0007 

-  .0043 

z( 

1) 

.0027 

.0016 

.0127 

.0027 

.0014 

.0009 

z( 

2) 

.0033 

.0012 

.0027 

.00^5 

.0029 

.0021 

z{ 

3) 

.0034 

-.0007 

.0014 

.0029 

.0065 

.0052 

z( 

4) 

.0023 

-.0043 

.0009 

.0021 

.0052 

.0105 

CORRELATION  MATRIX: 

A 

1.0000 

.6009 

.1181 

.2167 

.2049 

.1107 

B 

.6009 

1.0000 

.0968 

.1052 

-.0579 

-  .2769 
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z( 

1) 

.1181 

.0968 

1.0000 

.3287 

.1557 

.0769 

z( 

2) 

.2167 

.1052 

.3287 

1.0000 

.4862 

.2808 

z{ 

3) 

.2049 

-  .0579 

.1557 

.4862 

1.0000 

.6277 

z{ 

4) 

.1107 

-  .2769 

.0769 

.2808 

.6277 

1.0000 

AREA  = 

.8503 

STD. 

DEV.  (AREA)  = 

.0302 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.2379 

( 

.0932  , 

.4583 

) 

.010 

.3029 

( 

.1438  , 

.5125 

) 

.020 

.3819 

( 

.2161  , 

.5730 

) 

.030 

.4349 

( 

.2704  , 

.6117 

) 

.040 

.4756 

( 

.3146  , 

.6409 

) 

.050 

.5090 

.  ( 

.3521  , 

.6645 

) 

.060 

.5373 

( 

.3847  , 

.6846 

) 

.070 

.5620 

( 

.4137  , 

.7021 

) 

.080 

.5840 

( 

.4396  , 

.7177 

) 

.090 

.6037 

( 

.4632  , 

.7318 

) 

.100 

.6216 

( 

.4847  , 

.7447 

) 

.110 

.6380 

( 

.5044  , 

.7566 

) 

.120 

.6532 

( 

.5227  , 

.7676 

) 

.130 

.6673 

( 

.5396  , 

.7780 

) 

.140 

.6804 

( 

.5554  , 

.7877 

) 

.150 

.6927 

( 

.5702  , 

.7968 

) 

.200 

.7446 

( 

.6320  , 

.8360 

) 

.250 

.7852 

( 

.6796  , 

.8671 

) 

.300 

.8182 

( 

.7179  , 

.8926 

) 

.400 

.8692 

( 

.7768  , 

.9310 

) 

.500 

.9070 

( 

.8216  , 

.9576 

) 

.600 

.9360 

( 

.8583  , 

.9757 

) 

.700 

.9588 

( 

.8902  , 

.9876 

) 

.800 

.9766 

( 

.9197  , 

.9949 

) 

.900 

.9902 

( 

.9494  , 

.9988 

) 

.950 

.9956 

( 

.9664  , 

.9997 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .0917, 

.6069) 

(  .0629, 

.5449) 

( 

.1293, 

.6664) 

(  .2292, 

.7693) 

(  .1841, 

.7295) 

{ 

.2799, 

.8055) 

(  .5752, 

.9295) 

(  .5175, 

.9126) 

{ 

.6313, 

.9438) 

(  .9311, 

.9937) 

(  .8968, 

.9899) 

( 

.9559, 

.9962) 

3  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Appendix  HI  -  Reader  4 


False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 


(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


8/16/1998 


TIME  -  13: 


9:34 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAX 


IMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  04  Analog  Calc  for  345  cases  currently 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  261. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  84. 

RESPONSE  DATA: 

CATEGORY  1  2 

ACTUALLY  NEGATIVE  CASES  62.  150. 

ACTUALLY  POSITIVE  CASES  5.  24. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .042  .061  .188  .762  1. 

TPF:  .000  .476  .536  .655  .940  1. 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.0637  B=  .6587 

Z(K)=  -.714  .886  1.544  1.727 

LOGL=  -405.1970 
GOODNESS  OF  FIT  -- 
CHI  SQUARE=  1.7163  WITH  2  DEGREES  OF  FREEDOM,  P=  .4239 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.0396  B=  .6483 

Z(K)=  -.720  .903  1.507  1.711 

LOGL=  -404.9319 
GOODNESS  OF  FIT  -- 

CHI  SQUARE=  1.2423  WITH  2  DEGREES  OF  FREEDOM,  P=  .5373 


VARIANCE -COVARIANCE  MATRIX: 


1 

A 

.0254 

.0086 

.0032 

.0032 

.0023 

.0018 

B 

.0086 

.0097 

.0015 

-  .0007 

- .0035 

-  .0049 

z( 

1) 

.0032 

.0015 

.0072 

.0022 

.0014 

.0011 

Z( 

2) 

.0032 

-.0007 

.0022 

.0078 

.0065 

.0063 

z{ 

3) 

.0023 

- .0035 

.0014 

.0065 

.0133 

.0131 

z( 

4) 

.0018 

-.0049 

.0011 

.0063 

.0131 

.0169 

CORRELATION  MATRIX: 

A 

1.0000 

.5453 

.2340 

.2253 

.1276 

.0865 

B 

.5453 

1.0000 

.1783 

-.0816 

-.3069 

-.3842 

3  4  5 

33.  5.  11. 

10.  5.  40. 


000 

000 
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z( 

1) 

.2340 

.1783 

1.0000 

.2879 

.1424 

.1017 

z( 

2) 

.2253 

-  .0816 

.2879 

1.0000 

.6357 

.5459 

z( 

3) 

.1276 

-  .3069 

.1424 

.6357 

1.0000 

.8723 

z( 

4) 

.0865 

-  .3842 

.1017 

.5459 

.8723 

1.0000 

AREA  = 

.8085 

STD. 

DEV.  (AREA)  = 

.0319 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.2641 

( 

.1468  , 

.4164 

) 

.010 

.3195 

( 

.1971  , 

.4658 

) 

.020 

.3850 

( 

.2615  , 

.5217 

) 

.030 

.4285 

( 

.3065  , 

.5579 

) 

.040 

.4619 

( 

.3420  , 

.5854 

) 

.050 

.4892 

( 

.3715  , 

.6078 

) 

.060 

.5125 

( 

.3969  , 

.6270 

) 

.070 

.5329 

( 

.4193  , 

.6438 

) 

.080 

.5511 

( 

.4394  , 

.6589 

) 

.090 

.5676 

( 

.4576  , 

.6725 

) 

.100 

.5826 

( 

.4742  , 

,6850 

) 

.110 

.5965 

( 

.4896  , 

.6966 

) 

.120 

.6094 

( 

.5038  , 

.7074 

) 

.130 

.6214 

( 

.5171  , 

.7175 

) 

.140 

.6327 

( 

.5296  , 

.7271 

) 

.150 

.6434 

( 

.5414  , 

.7361 

) 

.200 

.6893 

( 

.5917  , 

.7752 

) 

.250 

.7266 

( 

.6321  , 

.8072 

) 

.300 

.7580 

( 

.6660  , 

.8342 

) 

.400 

.8094 

( 

.7213  , 

.8779 

) 

.500 

.8507 

( 

.7664  , 

.9118 

) 

.600 

.8856 

( 

.8056  , 

.9389 

) 

.700 

.9161 

( 

.8418  , 

.9605 

) 

.800 

.9435 

( 

.8772  , 

.9777 

) 

.900 

.9693 

( 

.9157  , 

.9910 

) 

.950 

.9824 

( 

.9396  , 

.9961 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0435, 

.4722) 

(  .0246, 

.4071) 

(  .0727, 

.5381) 

(  .0659, 

.5248) 

(  .0415, 

.4665) 

(  .1000, 

.5827) 

(  .1832, 

.6751) 

(  .1410, 

.6338) 

(  .2325, 

.7143) 

(  .7641, 

.9340) 

(  .7098, 

.9189) 

(  .8123, 

.9468) 

Do  you  want  to 

use  data  from  a 

previously 

created  input  file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


8/16/1998 
13:13: 44 

R  O  C  F  I  T  (IBM  VERSION  1.2) 


TIME  - 


MAX 


IMUM  LIKELIHOOD  E'S 
OF  A  BINORMAL  ROC 

FROM  RATING  DAT 


T  I  M  A  T  I 

CURVE 

A 


0  N 


DATA  DESCRIPTION:  Reader  04  Digital  Calc  for  345  cases  currently 
available 


DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  261. 
NO.  OF  ACTUALLY  POSITIVE  CASES  =  84. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  41.  171. 
CASES  12 .  30. 


3  4  5 

5.  11.  33. 

3.  8.  31. 


OBSERVED  OPERATING  POINTS: 

FPF:  .000  .126  .169  .188  .843  1.000 

TPF:  .000  .369  .464  .500  .857  1.000 


INITIAL  VALDES  OF  PARAMETERS: 

A=  .4642  B=  .6116 

Z{K)=  -1.006  .886  .960  1.143 

LOGL=  -387.7238 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALDES  OF  PARAMETERS: 

A=  .4682  B=  .6520 

Z{K)=  -.994  .856  .944  1.177 

LOGL=  -386.8398 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0179 

.0031 

.0033 

.0031 

.0031 

.0031 

B 

.0031 

.0087 

.0022 

-  .0015 

-.0017 

-  .0026 

z( 

1) 

.0033 

.0022 

.0087 

.0019 

.0017 

.0014 

z( 

2) 

.0031 

-.0015 

.0019 

.0077 

.0074 

.0069 

z{ 

3) 

.0031 

-.0017 

.0017 

.0074 

.0081 

.0076 

z( 

4) 

.0031 

-.0026 

.0014 

.0069 

.0076 

.0098 

, 

CORRELATION  MATRIX: 

A 

1.0000 

.2456 

.2685 

.2609 

.2530 

.2305 

B 

.2456 

1.0000 

.2481 

-  .1789 

-.2068 

-  .2795 
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z( 

1) 

.2685 

.2481 

1.0000 

.2307 

.2081 

.1524 

z( 

2) 

.2609 

-.1789 

.2307 

1.0000 

.9402 

.7982 

z( 

3) 

.2530 

-  .2068 

.2081 

.9402 

1.0000 

.8497 

z{ 

4) 

.2305 

-.2795 

.1524 

.7982 

.8497 

1.0000 

AREA  =  .6525  STD.  DEV. (AREA)  =  .0404 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.1128 

( 

.0453  , 

.2323 

) 

.010 

.1471 

( 

.0680  , 

.2720 

) 

.020 

.1918 

( 

.1014  , 

.3196 

) 

.030 

.2241 

( 

.1278  , 

.3519 

) 

.040 

.2503 

( 

.1504  , 

.3773 

) 

.050 

.2727 

( 

.1705  , 

.3986 

) 

.060 

.2926 

( 

.1888  , 

.4171 

) 

.070 

.3106 

( 

.2057  , 

.4337 

) 

.080 

.3270 

( 

.2214  , 

.4488 

) 

.090 

.3423 

( 

.2362  , 

.4626 

) 

.100 

.3566 

( 

.2503  , 

.4755 

) 

.110 

.3701 

( 

.2636  , 

.4877 

) 

.120 

.3829 

( 

.2763  , 

.4991 

) 

.130 

.3950 

( 

.2886  , 

.5100 

) 

.140 

.4066 

( 

.3003  , 

.5204 

) 

.150 

.4178 

( 

.3116  , 

.5303 

) 

.200 

.4679 

( 

.3631  , 

.5751 

) 

.250 

.5114 

( 

.4080  , 

.6141 

) 

) 

.300 

.5504 

( 

.4483  , 

.6491 

.400 

.6192 

( 

.5192  , 

.7117 

) 

.500 

.6802 

( 

.5815  , 

.7675 

) 

.600 

.7367 

( 

.6389  , 

.8188 

) 

.700 

.7910 

( 

.6946  , 

.8667 

) 

.800 

.8454 

( 

.7522  , 

.9119 

) 

.900 

.9039 

( 

.8189  , 

.9551 

) 

.950 

.9383 

( 

.8633  , 

.9765 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  PITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 
(  FPF  ,  TPF  ) 

( 

LOWER 
FPF  , 

BOUND 
TPF  ) 

( 

UPPER 
FPF  , 

BOUND 
TPF  ) 

(  .1196, 

.3824) 

( 

.0852, 

.3351) 

( 

.1629, 

.4315) 

(  .1725, 

.4414) 

( 

.1312, 

.3964) 

( 

.2214, 

.4872) 

(  .1961, 

.4643) 

( 

.1522, 

.4201) 

( 

.2470, 

.5088) 

(  .8399, 

.8679) 

( 

.7915, 

.8407) 

( 

.8803, 

.8917) 

Do  you  want  to 

use  data  from 

a  previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/N,  or  R  to  restart) 

DATE  - 


8/16/1998 
13:10:  8 

R  O  C  F  I  T  (IBM  VERSION  1.2) 


TIME  - 


MAXIMUM 
OF  A 
F 


L  I  K  E  L  I 
B  I  N  O  R  M 
ROM  RAT 


HOOD  E  S  T 
A  L  ROC  C 
I  N  G  DATA 


I  M  A  T  I  0  N 
U  R  V  E 


DATA  DESCRIPTION:  Reader  04  Analog  FAD  for  345  cases  currently 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT), 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  319. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  26. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  39.  102. 
CASES  1 .  4 . 


3  4  5 

48.  58.  72. 

4.  4.  13. 


OBSERVED  OPERATING  POINTS: 

FPF:  .000  .226  .408  .558  .878  1.000 

TPF:  .000  .500  .654  .808  ,962  1,000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  .6827  B=  .9378 

Z{K)=  -1.164  -.146  .234  .753 

LOGL=  -530.2149 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


PROCEDURE  CONVERGES  AFTER  3  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  ,6889  B=  .9298 

Z(K)=  -1.164  -.148  .239  .751 

LOGL=  -530.1941 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0551 

.0140 

.0030 

.0033 

.0034 

.0034 

B 

.0140 

.0530 

.0026 

.0008 

-  .0004 

-.0027 

z{ 

1) 

.0030 

.0026 

.0081 

.0027 

.0020 

.0014 

z( 

2) 

.0033 

.0008 

.0027 

-0049 

.0037 

.0027 

z( 

3) 

.0034 

-.0004 

.0020 

.0037 

.0049 

.0037 

z( 

4) 

.0034 

-.0027 

.0014 

.0027 

.0037 

.0060 

CORRELATION  MATRIX: 

A 

1.0000 

.2595 

.1423 

.2010 

.2029 

.1845 

B 

.2595 

1.0000 

.1253 

.0502 

-  .0257 

-.1524 
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z( 

1) 

.1423 

.1253 

1.0000 

.4323 

.3166 

.1980 

z( 

2) 

.2010 

.0502 

.4323 

1.0000 

.7422 

.4906 

z{ 

3) 

.2029 

-.0257 

.3166 

.7422 

1.0000 

.6695 

z( 

4) 

.1845 

-.1524 

.1980 

.4906 

.6695 

1.0000 

AREA  =  .6930  STD.  DEV.  (AREA)  =  .0585 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.0440 

( 

.0023  , 

.2833 

) 

.010 

.0702 

( 

.0061  , 

.3286 

) 

.020 

.1110 

( 

.0161  , 

.3822 

) 

.030 

.1445 

( 

.0278  , 

.4182 

) 

.040 

.1738 

( 

.0406  , 

.4463 

) 

.050 

.2002 

( 

.0542  , 

.4698 

) 

.060 

.2245 

( 

.0682  , 

.4901 

) 

.070 

.2471 

( 

.0826  , 

.5083 

) 

.080 

.2684 

( 

.0972  , 

.5247 

) 

.090 

.2884 

( 

.1120  , 

.  .5399 

) 

.100 

.3075 

( 

.1268  , 

.5540 

) 

.110 

.3257 

( 

.1417  , 

.5673 

) 

.120 

.3432 

( 

.1565  , 

.5798 

) 

.130 

.3  600 

( 

.1713  , 

.5917 

) 

.140 

.3761 

( 

.1860  , 

.6031 

) 

.150 

.3917 

( 

.2006  , 

.6140 

) 

.200 

.4627 

( 

.2714  , 

.6633 

) 

.250 

.5247 

( 

.3375  , 

.7065 

) 

.300 

.5799 

( 

.3982  , 

.7458 

) 

.400 

.6750 

( 

.5037  , 

.8154 

) 

.500 

.7545 

( 

.5904  , 

.8747 

) 

.600 

.8223 

( 

.6635  , 

.9231 

) 

.700 

.8802 

( 

.7280  , 

.9595 

) 

.800 

.9294 

( 

.7890  , 

.9838 

) 

.900 

.9700 

( 

.8539  , 

.9966 

) 

.950 

.9867 

( 

.8942  , 

.9993 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .2262, 

.4961) 

(  .1831, 

.4399) 

( 

.2745, 

.5524) 

(  .4054, 

.6795) 

(  .3530, 

.6323) 

( 

.4595, 

.7239) 

(  .5588, 

.7957) 

(  .5043, 

.7577) 

( 

.6122, 

.8299) 

(  .8777, 

.9617) 

(  .8382, 

.9459) 

( 

.9100, 

.9735) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  mm? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/N,  or  R  to  restart) 

DATE  - 

8/16/1998 

TIME  - 

13:14:15 

R  O  C  F  I  T  (IBM  VERSION  1.2)  ; 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 

DATA  DESCRIPTION:  Reader  04  Digital  FAD  for  345  cases  currently 
availcible 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  319. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  26. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  8.  62.  77.  93.  79. 

ACTUALLY  POSITIVE  CASES  0.  2.  3.  9.  12. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .248  .539  .781  .975  1.000 

TPF:  .000  .462  .808  .923  1.000  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  .6359  B=  .8031 

Z(K)=  -1.959  -.774  -.098  .682 

LOGL=  -497.0686 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .6813  B=  1.1201 

Z(K)=  -1.961  -.770  -.103  .684 

LOGL=  -496.0947 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0564 

.0161 

.0027 

.0037 

.0041 

.0042 

B 

.0161 

.0684 

.0028 

.0026 

.0009 

-  .0032 

z( 

1) 

.0027 

.0028 

.0222 

.0036 

.0019 

.0010 

z( 

2) 

.0037 

.0026 

.0036 

.0061 

.0032 

.0018 

z( 

3) 

.0041 

.0009 

.0019 

.0032 

.0049 

.0029 

z( 

4) 

.0042 

-.0032 

.0010 

.0018 

.0029 

.0058 

CORRELATION  MATRIX: 

A 

1.0000 

.2592 

.0763 

.2019 

.2479 

.2292 

B 

.2592 

1.0000 

.0714 

.1257 

.0474 

- .1625 
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z( 

1) 

.0763 

.0714 

1.0000 

.3066 

.1809 

.0885 

z( 

2) 

.2019 

.1257 

.3066 

1.0000 

.5897 

.3052 

z( 

3) 

.2479 

.0474 

.1809 

.5897 

1.0000 

.5402 

z( 

4) 

.2292 

-  .1625 

.0885 

.3052 

.5402 

1.0000 

AREA  =  .6750  STD.  DEV. (AREA)  =  .0554 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.0137 

( 

.0002 

t 

.1780 

) 

.010 

.0271 

( 

.0010 

i 

.2228 

) 

.020 

.0527 

( 

.0040 

t 

.2794 

) 

.030 

.0770 

( 

.0086 

t 

.3192 

) 

.040 

.1003 

( 

.0147 

/ 

.3510 

) 

.050 

.1227 

( 

.0221 

9 

.3781 

) 

.060 

.1444 

( 

.0306 

9 

.4019 

) 

.070 

.1655 

( 

.0400 

9 

.4234 

) 

.080 

.1860 

( 

.0503 

9 

.4430 

) 

.090 

.2059 

( 

.0613 

9 

.4612 

) 

.100 

.2253 

( 

.0730 

9 

.4782 

) 

.110 

.2442 

( 

.0852 

9 

.4943 

) 

.120 

.2627 

( 

.0979 

9 

.5095 

) 

.130 

.2808 

( 

.1110 

9 

.5241 

) 

.140 

.2985 

( 

.1244 

9 

.5380 

) 

.150 

.3157 

( 

.1382 

9 

.5514 

) 

.200 

.3970 

( 

.2096 

9 

.6123 

) 

.250 

.4706 

( 

.2821 

9 

.6660 

) 

.300 

.5376 

( 

.3524 

1 

.7149 

) 

.400 

.6547 

( 

.4794 

9 

.8017 

) 

.500 

.7522 

( 

.5854 

9 

.8743 

) 

.600 

.8326 

( 

.6736 

9 

.9305 

) 

.700 

.8976 

( 

.7497 

9 

.9688 

) 

.800 

.9478 

( 

.8191 

9 

.9902 

) 

.900 

.9829 

( 

.8885 

9 

.9987 

) 

.950 

.9942 

( 

.9279 

9 

.9998 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .2471, 

.4664) 

(  .2024, 

.4005) 

( 

.2967, 

.5332) 

(  .5411, 

.7872) 

(  .4866, 

.7401) 

( 

.5948, 

.8289) 

(  .7794, 

.9387) 

(  .7314, 

.9150) 

( 

.8220, 

.9568) 

(  .9751, 

.9980) 

(  .9524, 

.9946) 

( 

.9879, 

.9993) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader04  Analog  ROC  Type=Diag  N=345  Areas^.SOSO  STD  Area=.0286 


uoipBJj  aAjjjsod  9njx 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/n,  or  R  to  restart) 

DATE  - 

8/16/1998 

TIME  - 

13:10:37 

R  O  C  F  I  T  (IBM  VERSION  1.2)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  04  Analog  Diag  for  345  cases  currently 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  261. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  84. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  32.  128.  90.  9.  2. 

ACTUALLY  POSITIVE  CASES  0.  16.  36.  19.  13. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .008  .042  .387  .877  1.000 

TPF:  .000  .155  .381  .810  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3098  B=  .9665 

Z(K)=  -1.162  .287  1.727  2.425 

LOGL=  -405.3728 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.2437  B=  .9365 

Z(K)=  -1.177  .305  1.684  2.423 

LOGL=  -405.0512 

CHI-SQUARE  GOODNESS  OP  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0325 

.0127 

.0035 

.0051 

.0032 

-  .0021 

B 

.0127 

.0137 

.0016 

.0010 

-  .0061 

-.0146 

z{ 

1) 

.0035 

.0016 

.0100 

.0025 

.0009 

-  .0001 

z( 

2) 

.0051 

.0010 

.0025 

.0060 

.0033 

.0026 

z( 

3) 

.0032 

-.0061 

.0009 

.0033 

.0163 

.0181 

z( 

4) 

-.0021 

-.0146 

-.0001 

.0026 

.0181 

.0422 

CORREIiATION  MATRIX: 

A 

1.0000 

.6024 

.1954 

.3650 

.1395 

-.0565 

B 

.6024 

1.0000 

.1374 

.1144 

-.4077 

-.6077 
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z( 

1) 

.1954 

.1374 

1.0000 

.3171 

.0719 

-  .0032 

z( 

2) 

.3650 

.1144 

.3171 

1.0000 

.3365 

.1618 

z( 

3) 

.1395 

-.4077 

.0719 

.3365 

1.0000 

.6922 

z( 

4) 

-.0565 

-  .6077 

- .0032 

.1618 

.6922 

1.0000 

AREA  = 

.8180 

STD. 

DEV.  (AREA)  = 

.0286 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.1212 

( 

.0504  , 

.2429 

) 

.010 

.1748 

( 

.0865  , 

.3058 

) 

.020 

.2482 

( 

.1440  , 

.3831 

) 

.030 

.3022 

( 

.1908  , 

.4360 

) 

.040 

.3460 

( 

.2310  , 

.4774 

) 

.050 

.3832 

( 

.2664  , 

.5118 

) 

.060 

.4158 

( 

.2983  , 

.5414 

) 

.070 

.4449 

( 

.3273  , 

.5676 

) 

.080 

.4711 

( 

.3538  , 

.5910 

) 

.090 

.4952 

( 

.3784  , 

.6124 

) 

.100 

.5173 

( 

.4012  , 

.6319 

) 

.110 

.5378 

( 

.4226  , 

.6499 

) 

.120 

.5569 

( 

.4426  , 

.6667 

) 

.130 

.5749 

( 

.4614  , 

.6824 

) 

.140 

.5917 

( 

.4792  , 

.6971 

) 

,150 

.6076 

( 

.4960  , 

.7110 

) 

.200 

.6757 

( 

.5687  , 

.7699 

) 

.250 

.7298 

( 

.6272  , 

.8160 

) 

.300 

.7743 

( 

.6759  , 

.8531 

) 

.400 

.8430 

( 

.7534  , 

.9080 

) 

.500 

.8932 

( 

.8134  , 

.9449 

) 

.600 

.9306 

( 

.8620  , 

.9694 

) 

.700 

.9586 

( 

.9028  , 

.9850 

) 

.800 

.9789 

( 

.9379  *  , 

.9942 

) 

.900 

.9927 

( 

.9688  , 

.9988 

) 

.950 

.9973 

( 

.9834  , 

.9997 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .0077, 

.1526) 

(  .0024, 

.0804) 

( 

.0217, 

.2584) 

(  .0461, 

.3694) 

(  .0266, 

.2852) 

( 

.0758, 

.4605) 

(  .3800, 

.8309) 

(  .3236, 

.7925) 

( 

.4391, 

.8644) 

(  .8805, 

.9905) 

(  .8369, 

.9847) 

( 

.9151, 

.9943) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader04  Digital  ROC  Type=Diag  N=345  Area=.7737  STD  Area=.0332 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/N,  or  R  to  restart) 

DATE  - 


8/16/1998 


TIME  - 


13:14:46 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  04  Digital  Diag  for  345  cases  currently 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT) . 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  261. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  84. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  8.  95.  147.  7.  4. 

ACTUALLY  POSITIVE  CASES  0.  10.  46.  19.  9. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .015  .042  .605  .969  1.000 

TPF:  .000  .107  .333  .881  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .8961  B=  .8922 

Z(K)=  -1.872  -.267  1.727  2.162 

LOGL=  -353.0214 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.0263  B=  .9316 

Z{K)=  -1.884  -.255  1.654  2.344 

LOGL=  -349.7610 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0295 

.0096 

0034 

.0047 

.0048 

.0017 

B 

.0096 

.0119 

0018 

.0015 

-  .0053 

-  .0117 

z{ 

1) 

.0034 

.0018 

0235 

.0028 

.0006 

-.0003 

z( 

2) 

,0047 

.0015 

0028 

.0061 

.0021 

.0013 

z( 

3) 

.0048 

-.0053 

0006 

.0021 

.0162 

.0171 

z( 

4) 

.0017 

-.0117  - 

0003 

.0013 

.0171 

.0382 

CORRELATION  MATRIX : 

A 

1.0000 

.5136 

1288 

.3540 

.2217 

.0518 

B 

.5136 

1.0000 

1074 

.1806 

-.3839 

-.5493 
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z( 

1) 

.1288 

.1074 

1.0000 

.2374 

.0303 

-.0116 

z( 

2) 

.3540 

.1806 

.2374 

1.0000 

.2145 

.0828 

z( 

3) 

.2217 

-.3839 

.0303 

.2145 

1.0000 

.6897 

z( 

4) 

.0518 

-.5493 

-.0116 

.0828 

.6897 

1.0000 

AREA  =  .7737  STD.  DEV. (AREA)  =  .0332 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.0848 

( 

.0323  , 

.1844 

) 

.010 

.1269 

( 

.0576  , 

.2397 

) 

.020 

.1874 

( 

.1003  , 

.3103 

) 

.030 

.2339 

( 

.1368  , 

.3602 

) 

.040 

.2726 

( 

.1693  , 

.4001 

) 

.050 

.3063 

( 

.1988  , 

.4337 

) 

.060 

.3364 

( 

.2260  , 

.4631 

) 

.070 

.3636 

( 

.2512  , 

.4893 

) 

.080 

.3886 

( 

.2748  , 

.5130 

) 

.090 

.4118 

( 

.2970  , 

.5348 

) 

.100 

.4334 

( 

.3179  , 

.5549 

) 

.110 

.4536 

( 

.3378  , 

.5737 

) 

.120 

.4727 

( 

.3566  , 

.5912 

) 

.130 

.4908 

( 

.3746  , 

.6077 

) 

.140 

.5079 

( 

.3918  , 

.6234 

) 

.150 

.5242 

( 

.4083  , 

.6381 

) 

.200 

.5958 

( 

.4815  , 

.7024 

) 

.250 

.6548 

( 

.5430  , 

.7544 

) 

.300 

.7048 

( 

.5960  , 

.7976 

) 

.400 

.7854 

( 

.6838  , 

.8650 

) 

.500 

.8476 

( 

.7548  , 

.9135 

) 

.600 

.8965 

( 

.8145  , 

.9484 

) 

.700 

.9350 

( 

.8663  , 

.9726 

) 

.800 

.9649 

( 

.9122  , 

.9883 

) 

-900 

.9868 

( 

.9544  , 

.9970 

) 

.950 

.9947 

( 

.9750  , 

.9992 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  {  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(  .0095,  .1235)  (  .0032,  .0649)  (  .0249,  .2116) 

(  .0491,  .3035)  {  .0285,  .2277)  (  .0801,  .3889) 

(  .6006,  .8968)  (  .5407,  .8689)  (  .6582,  .9201) 

(  .9702,  .9973)  (  .9433,  .9938)  (  .9855,  .9989) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


11/18/1998 

14:44:18 

R  O  C  F  I  T  (IBM  VERSION  1.2) 


TIME  - 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  05  Analog  Mass  resp  for  329  cases 
available 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  279. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  50. 

RESPONSE  DATA: 

CATEGORY  1  2 

ACTUALLY  NEGATIVE  CASES  165.  81. 

ACTUALLY  POSITIVE  CASES  1.  8. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .043  .079  .118  .409  1.000 

TPF:  .000  .540  .800  .820  .980  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  2.3795  B=  1.2337 

Z(K)=  .231  1.184  1.413  1.717 

LOGL=  -354.2874 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  2.5170  B=  1.3583 

Z(K)=  .233  1.175  1.354  1.760 

LOGL=  -352.6045 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


3  4  5 

11.  10.  12. 

1.  13.  27. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.2116 

.1258 

.0077 

.0095 

.0059 

-.0087 

B 

.1258 

.0947 

.0025 

-  .0008 

-.0047 

-  .0182 

z( 

1) 

.0077 

.0025 

.0057 

.0033 

.0030 

.0024 

z( 

2) 

.0095 

-.0008 

.0033 

.0091 

.0085 

.0079 

z( 

3) 

.0059 

-.0047 

.0030 

.0085 

.0106 

.0101 

z( 

4) 

-.0087 

-.0182 

.0024 

.0079 

.0101 

.0177 

CORRELATION  MATRIX: 

A 

1.0000 

.8885 

.2207 

.2180 

.1257 

-.1413 

B 

.8885 

1.0000 

.1077 

-  .0288 

-  .1474 

-  .4452 
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z( 

1) 

.2207 

.1077 

1.0000 

.4622 

.3917 

.2427 

z( 

2) 

.2180 

-.0288 

.4622 

1.0000 

.8720 

.6195 

z( 

3) 

.1257 

-.1474 

.3917 

.8720 

1,0000 

.7397 

z( 

4) 

- .1413 

-.4452 

.2427 

.6195 

.7397 

1.0000 

AREA  =  .9322  STD.  DEV. (AREA)  =  .0167 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

/ 

UPPER  BOUND) 

.005 

.1630 

( 

.0328 

/ 

.4508 

) 

.010 

.2600 

( 

.0847 

/ 

.5347 

) 

.020 

.3923 

( 

.1907 

/ 

.6289 

) 

.030 

.4847 

( 

.2845 

/ 

.6890 

) 

.040 

.5551 

( 

.3642 

/ 

.7338 

) 

.050 

.6111 

( 

.4312 

/ 

.7697 

) 

.060 

.6572 

( 

.4877 

# 

.7997 

) 

.070 

.6957 

( 

.5354 

/ 

.8252 

) 

.080 

.7285 

( 

.5761 

t 

.8472 

) 

.090 

.7567 

( 

.6110 

/ 

.8664 

) 

.100 

.7811 

( 

.6411 

/ 

.8831 

) 

.110 

.8026 

( 

.6674 

/ 

.8978 

) 

.120 

.8214 

( 

.6904 

/ 

.9106 

) 

.130 

.8382 

( 

.7108 

# 

.9219 

) 

.140 

.8530 

( 

.7290 

/ 

.9318 

) 

.150 

.8663 

( 

.7453 

/ 

.9405 

) 

.200 

.9153 

( 

.8070 

/ 

.9700 

) 

.250 

.9453 

( 

.8484 

/ 

.9851 

) 

.300 

.9645 

( 

,8785 

/ 

.9927 

) 

.400 

.9851 

( 

.9199 

# 

.9984 

) 

.500 

.9941 

( 

.9469 

# 

.9997 

) 

.600 

.9979 

( 

.9657 

f 

1.0000 

) 

.700 

.9994 

( 

.9792 

/ 

1.0000 

) 

.800 

.9999 

( 

.9889 

t 

1.0000 

) 

.900 

1.0000 

( 

.9957 

/ 

1.0000 

) 

.950 

1.0000 

( 

.9982 

t 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0392, 

.5505) 

(  .0217, 

.4100) 

(  .0670, 

.6850) 

(  .0879, 

.7510) 

(  .0599, 

.6570) 

(  .1245, 

.8292) 

(  .1200, 

.8214) 

(  .0867, 

.7478) 

(  .1614, 

.8798) 

(  .4077, 

.9861) 

(  .3513, 

.9772) 

(  .4662, 

.9918) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


11/18/1998 
14 : 45 : 26 

R  O  C  F  I  T  (IBM  VERSION  1.2) 


TIME  - 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  05  Analog  calc  reap  for  329  cases 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  252. 
NO.  OF  ACTUALLY  POSITIVE  CASES  =  77. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  174.  36. 
CASES.  13 .  3 . 


3  4  5 

3.  9.  30. 

1.  5.  55. 


OBSERVED  OPE^TING  POINTS: 

FPF:  .000  .119  .155  .167  .310  1.000 

TPF:  .000  .714  .779  .792  .831  1.000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2409  B=  .5071 

Z(K)=  .497  .967  1.016  1.180 

LOGL=  -313.2130 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.2739  B=  .5885 

Z{K)=  .500  .939  .992  1.193 

LOGL=  -312.6219 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0578 

.0361 

.0049 

.0024 

.0020 

.0001 

B 

.0361 

.0387 

.0019 

-.0018 

-.0025 

-.0052 

z( 

1) 

.0049 

.0019 

.0068 

.0053 

,0051 

.0046 

z{ 

2) 

.0024 

-.0018 

.0053 

.0084 

.0082 

.0077 

z( 

3) 

.0020 

-.0025 

.0051 

.0082 

.0087 

.0082 

z( 

4) 

.0001 

-.0052 

.0046 

.0077 

.0082 

.0106 

CORRELATION  MATRIX: 

A 

1.0000 

.7643 

.2480 

.1082 

.0872 

.0023 

B 

.7643 

1.0000 

.1142 

-  .1025 

-  .1334 

-.2549 

z( 

1) 

.2480 

.1142 

1.0000 

.7002 

.6664 

.5436 
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z(  2) 
Z(  3) 
Z(  4) 


.1082  -.1025  .7002 
.0872  -.1334  .6664 
.0023  -.2549  .5436 


1.0000  .9600  .8175 

.9600  1.0000  .8549 

.8175  .8549  1.0000 


AREA  =  .8639 


STD.  DEV. (AREA)  =  .0324 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.4043 

( 

.1725  , 

.6771 

) 

.010 

.4620 

( 

.2382  , 

.6989 

) 

.020 

.5259 

( 

.3218  , 

.7233 

) 

.030 

.5662 

( 

.3790  , 

.7394 

) 

.040 

.5961 

( 

.4230  , 

.7520 

) 

.050 

.6201 

( 

.4588  , 

.7626 

) 

.060 

.6401 

( 

.4888  , 

.7720 

) 

.070 

.6573 

( 

.5146  , 

.7805 

) 

.080 

.6725 

( 

.5370  , 

.7884 

) 

.090 

.6861 

( 

.5568  , 

.7958 

) 

.100 

.6983 

( 

.5744  , 

.8028 

) 

.110 

.7095 

( 

.5901  , 

.8095 

) 

.120 

.7198 

( 

.6043  , 

.8160 

) 

.130 

.7294 

( 

.6172  , 

.8222 

) 

.140 

.7383 

( 

.6289  , 

.8282 

) 

.150 

.7466 

( 

.6396  , 

.8341 

) 

.200 

.7819 

( 

.6818  , 

.8610 

) 

.250 

.8098 

( 

.7112  , 

.8844 

) 

.300 

.8329 

( 

.7333  , 

.9046 

) 

.400 

.8697 

( 

.7652  , 

.9366 

) 

.500 

.8987 

( 

.7889  , 

.9595 

) 

.600 

.9226 

( 

.8088  / 

.9757 

) 

.700 

.9432 

( 

.8272  , 

.9868 

) 

.800 

.9616 

( 

.8462  , 

.9941 

) 

.900 

.9787 

( 

.8690  , 

.9983 

) 

.950 

.9875 

( 

.8854  , 

.9995 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .1165, 

.7163) 

(  .0816, 

.6748) 

( 

.1608, 

.7551) 

(  .1606, 

.7549) 

(  .1199, 

.7198) 

( 

.2093, 

.7875) 

(  .1739, 

.7647) 

(  .1317, 

.7310) 

( 

.2239, 

.7959) 

(  .3087, 

.8364) 

(  .2541, 

.8118) 

( 

.3678, 

.8588) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

The  input  file  (  )  CANNOT  be  found  in  your  current 

subdirectory I 

enter  INPUT  file  name: 
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The  input  file  (  )  CANNOT  be  found  in  your  current 

siibdirectory  I 

enter  INPUT  file  name: 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE 

11/18/1998 

TIME  - 

14:44:18 

R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  05  Analog  fad  resp  for  329  cases 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  307. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  22. 

RESPONSE  DATA: 

CATEGORY  1  2  3  4  5 

ACTUALLY  NEGATIVE  CASES  244.  37.  6.  9.  11. 

ACTUALLY  POSITIVE  CASES  14.  3.  1.  1.  3. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .036  .065  .085  .205  1.000 

TPF:  .000  .136  .182  .227  .364  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .2908  B=  .7736 

Z(K)=  .823  1.374  1.513  1.802 

LOGL=  -250.8848 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  3  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .2825  B=  .7658 

Z(K)=  .823  1.372  1.518  1.801 

LOGL=  -250.8741 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1957 

.1132 

0066 

.0010 

-  .0009 

-.0054 

B 

.1132 

.1017 

0020 

-.0037 

-.0056 

-.0101 

z( 

1) 

.0066 

.0020 

0066 

.0050 

.0047 

.0041 

Z( 

2) 

.0010 

-.0037 

0050 

.0102 

.0097 

.0089 

z( 

3) 

-.0009 

-.0056 

0047 

.0097 

.0121 

.0112 

z( 

4) 

- .0054 

-.0101 

0041 

.0089 

.0112 

.0179 

CORRELATION  MATRIX: 

A 

1.0000 

.8022 

1849 

.0223 

-  .0183 

-.0907 

B 

.8022 

1.0000 

0758 

-.1134 

-.1586 

-.2371 

z( 

1) 

.1849 

.0758  1. 

0000 

.6082 

.5230 

.3806 
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z( 

2) 

.0223 

-.1134 

.6082 

1.0000 

.8716 

.6580 

z( 

3) 

-.0183 

-.1586 

.5230 

.8716 

1.0000 

.7572 

z( 

4) 

-.0907 

-.2371 

.3806 

.6580 

.7572 

1.0000 

AREA  =  .5887  STD.  DEV.  (AREA)  =  .1261 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.0455 

( 

.0031  , 

.2614 

) 

.010 

.0669 

( 

.0078  , 

.2808 

) 

.020 

.0984 

( 

.0190  , 

.3062 

) 

.030 

.1234 

( 

.0311  , 

.3257 

) 

.040 

.1449 

( 

.0434  , 

.3429 

) 

.050 

.1642 

( 

.0556  , 

.3588 

) 

.060 

.1818 

( 

.0674  , 

.3740 

) 

.070 

.1983 

( 

.0788  , 

.3888 

) 

.080 

.2137 

( 

.0897  , 

.4034 

) 

.090 

.2283 

( 

.1000  , 

.4179 

) 

.100 

.2423 

( 

.1098  , 

.4323 

) 

.110 

.2556 

( 

.1191  , 

.4466 

) 

.120 

.2685 

( 

.1279  , 

.4609 

) 

.130 

.2809 

( 

.1361  , 

.4751 

) 

.140 

.2930 

( 

.1439  , 

.4893 

) 

.150 

.3046 

( 

.1513  , 

.5035 

) 

.200 

.3587 

( 

.1823  , 

.5725 

) 

.250 

.4076 

( 

.2063  , 

.6376 

) 

.300 

.4527 

( 

.2255  , 

.6972 

) 

.400 

.5354 

( 

.2555  , 

.7981 

) 

.500 

.6112 

( 

.2795  , 

.8748 

) 

.600 

.6830 

( 

.3008  , 

.9298 

) 

.700 

.7530 

( 

.3218  , 

.9664 

) 

.800 

.8230 

( 

.3447  , 

.9879 

) 

.900 

.8969 

( 

.3746  , 

.9978 

) 

.950 

.9385 

( 

.3984  , 

.9996 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .0359, 

.1364) 

(  .0195, 

.0972) 

( 

.0619, 

.1852) 

(  .0646, 

.1895) 

(  .0415, 

.1480) 

( 

.0965, 

.2375) 

(  .0851, 

.2212) 

(  .0582, 

.1788) 

( 

.1203, 

.2689) 

(  .2052, 

.3640) 

(  .1630, 

.3193) 

( 

.2532, 

.4105) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

The  input  file  (  )  CANNOT  be  found  in  your  current 

subdirectory ! 

enter  INPUT  file  name: 
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)  CANNOT  be  found  in  your  current 


The  input  file  ( 
subdirectory! 
enter  INPUT  file  name: 

The  input  file  (  )  CANNOT  be  found  in  your  current 

sxabdirectory ! 

enter  INPUT  file  name: 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 

11/18/1998 

TIME  - 

14:44:18 

R  O  C  F  I  T  (IBM  VERSION  1.2)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  05  Analog  diag  resp  for  329  cases 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  240. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  89. 

RESPONSE  DATA: 

CATEGORY  1  2  3  4  5 

ACTUALLY  NEGATIVE  CASES  105.  85.  22.  25.  3. 

ACTUALLY  POSITIVE  CASES  5.  21.  12.  27.  24. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .013  .117  .208  .563  1.000 

TPF:  .000  .270  .573  .708  .944  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  1.3631  B=  .9165 

Z(K)=  -.157  .812  1.192  2.242 

LOGL=  -430.6153 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.3603  B=  .9327 

Z(K)=  -.163  .831  1.215  2.144 

LOGL=  -430.1785 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0349 

.0159 

0052 

.0053 

.0041 

-.0029 

B 

.0159 

.0166 

0020 

-  .0005 

-.0029 

-.0141 

Z{ 

1) 

.0052 

.0020 

0066 

.0035 

.0028 

.0013 

z{ 

2) 

.0053 

-  .0005 

0035 

.0078 

.0068 

.0063 

z( 

3) 

.0041 

-.0029 

0028 

.0068 

.0102 

.0104 

Z{ 

4) 

-.0029 

-  .0141 

0013 

.0063 

.0104 

.0321 

CORRELATION  MATRIX: 

A 

1.0000 

.6596 

3466 

.3211 

.2196 

-.0860 

B 

.6596 

1.0000 

1888 

-  .0431 

-  .2266 

-.6133 

z( 

1) 

.3466 

.1888  1. 

0000 

.4857 

.3442 

.0883 
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z{ 

2) 

.3211 

-  .0431 

.4857 

1.0000 

.7687 

.3964 

z( 

3) 

.2196 

-.•2266 

.3442 

.7687 

1.0000 

.5776 

z{ 

4) 

-.0860 

-  .6133 

.0883 

.3964 

.5776 

1.0000 

AREA  = 

.8401 

STD. 

DEV.  (AREA)  = 

.0258 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


005 

.1486 

( 

.0624  , 

.2912 

) 

010 

.2090 

( 

.1054  , 

.3564 

) 

020 

.2892 

( 

.1721  , 

.4344 

) 

030 

.3467 

( 

.2251  , 

.4866 

) 

040 

.3925 

( 

.2698  , 

.5269 

) 

050 

.4309 

( 

.3087  , 

.5601 

) 

060 

.4641 

( 

.3431  , 

.5885 

) 

070 

.4935 

( 

.3740  , 

.6135 

) 

080 

.5198 

( 

.4021  , 

.6358 

) 

090 

.5436 

( 

.4277  , 

.6559 

) 

100 

.5655 

( 

.4513  , 

.6744 

) 

110 

.5856 

( 

.4731  , 

.6914 

) 

120 

.6042 

( 

.4934  , 

.7071 

) 

130 

.6216 

( 

.5124  , 

.7218 

) 

140 

.6378 

( 

.5301  , 

.7356 

) 

150 

.6531 

( 

.5468  , 

.7485 

) 

200 

.7175 

( 

.6174  / 

.8030 

) 

250 

.7678 

( 

.6726  , 

.8452 

) 

300 

.8083 

( 

.7175  , 

.8786 

) 

400 

.8696 

( 

.7872  , 

.9268 

) 

500 

.9131 

( 

.8399  , 

.9579 

) 

600 

.9448 

( 

.8820  , 

.9776 

) 

700 

.9678 

( 

.9169  , 

.9896 

) 

800 

.9840 

( 

.9468  , 

.9963 

) 

900 

.9947 

( 

.9730  , 

.9993 

) 

950 

.9981 

( 

.9855  , 

.9998 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OP  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  ,0160, 

,2614) 

(  .0063, 

.1668) 

( 

.0365, 

.3777) 

(  .1122, 

.5898) 

(  .0789, 

.5171) 

( 

.1545, 

.6596) 

(  .2030, 

.7209) 

(  .1577, 

.6642) 

( 

.2553, 

.7724) 

(  .5649, 

.9348) 

(  .5019, 

.9138) 

( 

.6264, 

.9516) 

Do  you  want  to 

use  data  from  a 

previously 

created 

input 

file  for  the 

next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

The  input  file  {  )  CANNOT  be  found  in  your  current 

siibdirectory  I 

enter  INPUT  file  name: 
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)  CANNOT  be  found  in  your  current 


The  input  file  ( 
subdirectory I 
enter  INPUT  file  name: 

The  input  file  (  )  CANNOT  be  found  in  your  current 

siibdirectory  I 

enter  INPUT  file  name: 

The  input  file  (  )  CANNOT  be  found  in  your  current 

subdirectory I 

enter  INPUT  file  name: 
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Reader06  Analog  (191  cases 
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ReaderOG  Analog  ROC  Type=Mass  N=191  Area=.8958  STD  Area=.0458 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  - 


8/28/1998 


TIME  - 


8:49:  3 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  06  Analog  Mass  resp  for  191  cases 
available. 

DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  171. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  20. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  33.  118.  11.  4.  5. 

ACTUALLY  POSITIVE  CASES  1.  2.  2.  6.  9. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .029  .053  .117  .807  1.000 

TPF:  .000  .450  .750  .850  .950  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3051  B=  .5187 

Z{K)=  -.867  1.190  1.620  1.893 

LOGL=  -194.4213 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.5905  B=  .7737 

Z(K)=  -.853  1.150  1.574  2.059 

LOGL=  -191.3606 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.2014 

.0878 

.0051 

.0093 

.0050 

-  .0077 

B 

.0878 

.0613 

.0020 

.0001 

- .0056 

-  .0193 

z( 

1) 

.0051 

.0020 

.0120 

.0026 

.0021 

.0014 

z( 

2) 

.0093 

.0001 

.0026 

.0148 

.0126 

.0111 

z( 

3) 

.0050 

-.0056 

.0021 

.0126 

.0226 

.0208 

z( 

4) 

-.0077 

-.0193 

.0014 

.0111 

.0208 

.0449 

CORRELATION  MATRIX: 

A 

1.0000 

.7905 

.1033 

.1707 

.0739 

-.0804 

B 

.7905 

1.0000 

.0752 

.0030 

-  .1507 

-  .3677 
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z( 

1) 

.1033 

.0752 

1.0000 

.1956 

.1254 

.0606 

z( 

2) 

.1707 

.0030 

.1956 

1.0000 

.6864 

.4283 

z( 

3) 

.0739 

-  .1507 

.1254 

.6864 

1.0000 

.6538 

z( 

4) 

-.0804 

-.3677 

.0606 

.4283 

.6538 

1.0000 

AREA  =  .8958  STD.  DEV.  (AREA)  =  .0458 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.3436 

( 

.1198  , 

.6445 

) 

.010 

.4169 

{ 

.1837  , 

.6850 

) 

.020 

.5005 

( 

.2689  , 

.7319 

) 

.030 

.5537 

( 

.3278  , 

.7630 

) 

.040 

.5932 

( 

.3727  , 

.7870 

) 

.050 

.6246 

( 

.4087  , 

.8068 

) 

.060 

.6507 

( 

.4386  , 

.8236 

) 

.070 

.6731 

( 

.4639  , 

.8383 

) 

.080 

.6926 

( 

.4858  , 

.8513 

) 

.090 

.7099 

( 

.5050  , 

.8629 

) 

.100 

.7254 

{ 

.5220  , 

.8734 

) 

.110 

.7394 

{ 

.5373  , 

.8828 

) 

.120 

.7522 

( 

.5510  , 

.8915 

) 

.130 

.7639 

( 

.5636  , 

.8993 

) 

.140 

.7748 

{ 

.5751  , 

.9066 

) 

.150 

.7848 

( 

.5857  , 

.9132 

) 

.200 

.8263 

{ 

.6290  , 

.9394 

) 

.250 

.8574 

{ 

.6614  , 

.9574 

) 

.300 

.8820 

{ 

.6875  , 

.9700 

) 

.400 

.9185 

{ 

.7286  , 

.9854 

) 

.500 

.9441 

( 

.7614  , 

.9932 

) 

.600 

.9630 

{ 

.7901  , 

.9972 

) 

.700 

.9770 

( 

.8171  , 

.9990 

) 

.800 

.9875 

( 

.8446  , 

.9997 

) 

.900 

.9951 

( 

.8768  , 

1.0000 

) 

.950 

.9979 

( 

.8990  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  {  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(  .0197,  .4989)  (  .0067,  .3728)  (  .0501,  .6250) 

(  .0577,  .6452)  (  .0308,  .5575)  (  .1003,  .7258) 

(  .1250,  .7582)  {  .0824,  .6970)  (  .1809,  .8119) 

(  .8030,  .9878)  (  .7382,  .9814)  (  .8571,  .9922) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the 
next  run? 

(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  8/28/1998 

TIME  -  8:49:  3 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  06  Digital  Mass  resp  for  191  cases  available. 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  171. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  20. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .029  .041  .082  .895  1.000 

TPF:  .000  .250  .500  .800  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.1886  B=  .6962 
Z(K)=  -1.252  1.393  1.740  1.893 

LOGL=  -155.8947 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  6  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  4.9718  B=  2.9439 
Z(K)=  -1.252  1.393  1.708  1.909 

LOGL=  -145.9707 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

18.  139.  7.  2.  5. 

0.  4.  6.  5.  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

3.5380 

2.1296 

.0082 

.0865 

-  .0338 

-  .1326 

B 

2.1296 

1.3725 

.0020 

.0215 

-.0616 

-.1269 

z{ 

1) 

.0082 

.0020 

.0166 

.0018 

.0016 

.0015 

z( 

2) 

.0865 

.0215 

.0018 

.0192 

.0169 

.0155 

z( 

3) 

-.0338 

-.0616 

.0016 

.0169 

.0258 

.0277 

Z{ 

4) 

-.1326 

- .1269 

.0015 

.0155 

.0277 

.0376 

CORRELATION  MATRIX: 

A  1.0000  .9664  .0339  .3315  -.1119  -.3633 
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B 

.9664 

1.0000 

.0135 

.1323 

-.3272 

-.5581 

Z( 

1) 

.0339 

.0135 

1.0000 

.1023 

.0775 

.0588 

z( 

2) 

.3315 

.1323 

.1023 

1.0000 

.7572 

.5747 

z( 

3) 

-.1119 

-.3272 

.0775 

.7572 

1.0000 

.8888 

z( 

4) 

-.3633 

-  .5581 

.0588 

.5747 

.8888 

1.0000 

AREA  =  .9451  STD.  DEV. (AREA)  =  .0173 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

/ 

UPPER  BOUND) 

.005 

.0045 

( 

.0000 

f 

.4697 

) 

.010 

.0302 

( 

.0000 

f 

.5551 

) 

.020 

.1411 

( 

.0051 

f 

.6619 

) 

.030 

.2856 

( 

.0376 

t 

.7410 

) 

.040 

.4273 

( 

.1083 

f 

.8076 

) 

.050 

.5511 

( 

.1996 

t 

.8643 

) 

.060 

.6531 

( 

.2899 

t 

.9101 

) 

.070 

.7345 

( 

.3681 

t 

.9440 

) 

.080 

.7981 

( 

.4323 

t 

.9671 

) 

.090 

.8471 

( 

.  .4842 

t 

.9816 

) 

.100 

.8846 

( 

.5265 

t 

.9901 

) 

.110 

.9132 

( 

.5614 

r 

.9949 

) 

.120 

.9348 

( 

.5909 

f 

-9974 

) 

.130 

.9511 

( 

.6161 

t 

.9987 

) 

.140 

.9634 

( 

.6380 

t 

.9994 

) 

.150 

.9726 

( 

.6573 

/ 

.9997 

) 

.200 

.9937 

( 

.7286 

/ 

1.0000 

) 

.250 

.9986 

( 

.7766 

/ 

1.0000 

) 

.300 

.9997 

( 

.8123 

/ 

1.0000 

) 

.400 

1.0000 

( 

.8639 

# 

1.0000 

) 

.500 

1.0000 

( 

.9006 

1 

1.0000 

) 

.600 

1.0000 

( 

.9286 

t 

1.0000 

) 

.700 

1.0000 

( 

.9510 

r 

1.0000 

) 

.800 

1.0000 

( 

.9695 

f 

1.0000 

) 

.900 

1.0000 

( 

.9850 

f 

1.0000 

) 

.950 

1.0000 

( 

.9921 

f 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 


{  FPF  ,  TPF  )  ( 

(  .0281,  .2585)  ( 

(  .0439,  .4781)  { 

(  .0817,  .8077)  ( 

{  .8947,  1.0000)  ( 


LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

.0110, 

.0386) 

(  .0632, 

.6814) 

.0216, 

.1631) 

(  .0819, 

.8084) 

.0479, 

.5277) 

(  .1310, 

.9525) 

.8412, 

1.0000) 

(  .9338, 

1.0000) 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name; 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC 
(Y/N,  or  R  to  restart) 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


file  for  the  next  run? 


curve? 

DATE  -  8/28/1998 

TIME  -  8:49;  3 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  06  Analog  Calc  resp  for  191  cases  available. 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
{E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  153. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  38. 

RESPONSE  DATA; 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  35.  99.  9.  6.  4. 

ACTUALLY  POSITIVE  CASES  3.  6.  2.  3.  24. 

OBSERVED  OPERATING  POINTS; 

FPF:  .000  .026  .065  .124  .771  1.000 

TPF:  .000  .632  .711  .763  .921  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  1.1337  B=  .3923 

Z(K)=  -.743  1.154  1.512  1.941 

LOGL=  -197.5969 

CHI-SQUARE  GOODNESS  OF  PIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  3  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.1422  B=  .4062 

Z(K)=  -.740  1.146  1.508  1.968 

LOGL=  -197.5536 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0583 

.0151 

.0035 

.0035 

.0032 

.0019 

B 

.0151 

.0130 

.0015 

-.0010 

-.0028 

-.0074 

z( 

1) 

.0035 

.0015 

.0125 

.0031 

.0025 

.0018 

z( 

2) 

.0035 

-.0010 

.0031 

.0164 

.0143 

.0129 

z( 

3) 

.0032 

-.0028 

.0025 

.0143 

.0233 

.0213 

z( 

4) 

.0019 

-.0074 

.0018 

.0129 

.0213 

.0430 

CORRELATION  MATRIX: 

A 

1.0000 

.5489 

.1313 

.1134 

.0856 

.0372 
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B 

.5489 

1.0000 

.1165 

-.0705 

-.1631 

-.3141 

z{ 

1) 

.1313 

.1165 

1.0000 

.2186 

.1483 

.0757 

z( 

2) 

.1134 

-.0705 

.2186 

1.0000 

.7318 

.4854 

z{ 

3) 

.0856 

-.1631 

.1483 

.7318 

1.0000 

.6728 

Z{ 

4) 

.0372 

-.3141 

.0757 

,4854 

.6728 

1.0000 

AREA  =  .8550  STD,  DEV.  (AREA)  =  .0464 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.5381 

( 

.3406  , 

.7264 

) 

.010 

.5781 

( 

.3911  , 

.7488 

) 

.020 

.6209 

( 

.4462  , 

.7736 

) 

.030 

.6473 

( 

.4804  , 

.7896 

) 

.040 

.6667 

( 

.5055  , 

.8018 

) 

.050 

.6822 

( 

.5254  , 

.8117 

) 

.060 

.6952 

( 

.5419  , 

.8201 

) 

.070 

.7063 

( 

.5560  , 

.8275 

) 

.080 

.7161 

( 

.5682  , 

.8342 

) 

.090 

.7249 

( 

.5792  , 

.8402 

) 

.100 

.7329 

( 

.5890  , 

.8457 

) 

.110 

.7402 

( 

•  5979  , 

.8508 

) 

.120 

.7469 

( 

.6061  , 

.8556 

) 

.130 

.7532 

( 

.6136  , 

.8600 

) 

,140 

.7591 

( 

.6207  , 

.8642 

) 

.150 

.7646 

( 

.6272  , 

.8682 

) 

.200 

.7883 

( 

.6548  , 

.8854 

) 

.250 

.8074 

( 

.6765  , 

.8995 

) 

.300 

.8236 

( 

.6945  , 

.9115 

) 

.400 

.8507 

( 

.7239  , 

.9311 

) 

.500 

.8733 

( 

.7482  , 

.9469 

) 

.600 

.8934 

( 

.7698  , 

.9601 

) 

.700 

.9123 

( 

.7906  , 

.9714 

) 

.800 

.9311 

( 

.8123  , 

.9813 

) 

.900 

.9518 

( 

.8385  , 

.9903 

) 

.950 

.9649 

( 

.8574  , 

.9947 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0245, 

.6341) 

(  .0088, 

.5706) 

(  .0592, 

.6942) 

(  .0657, 

.7018) 

(  .0353, 

.6583) 

(  .1133, 

.7425) 

(  .1259, 

.7507) 

(  .0812, 

.7173) 

(  .1854, 

,7819) 

(  .7705, 

.9255) 

(  .6988, 

.9121) 

(  .8314, 

.9372) 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  8/28/1998 

TIME  -  8:49:  3 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  06  Digital  Calc  resp  for  191  cases  available. 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  153. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  38. 

RESPONSE  DATA; 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .007  .059  .111  .902  1.000 

TPF:  .000  .263  .342  .421  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.0463  B=  .8055 

Z(K)=  -1.293  1.221  1.565  2.482 
LOGL=  -161.2745 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  6  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS; 

A=  .9989  B=  .8272 

Z{K)=  -1.318  1.273  1.587  2.120 
LOGL=  -159.0667 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

15.  121.  8.  8.  1. 

0.  22.  3.  3.  10. 


VARIANCE-COVARIANCE  MATRIX: 


A 

.0781 

.0301 

.0083 

.0103 

.0076 

.0004 

B 

.0301 

.0265 

.0042 

-  .0021 

-  .0059 

-  .0156 

z{ 

1) 

.0083 

.0042 

.0197 

.0024 

.0017 

.0002 

Z{ 

2) 

.0103 

-.0021 

.0024 

.0182 

.0168 

.0160 

z( 

3) 

.0076 

-.0059 

.0017 

.0168 

.0244 

.0238 

z{ 

4) 

.0004 

-.0156 

.0002 

.0160 

.0238 

.0484 

CORRELATION  MATRIX: 

A  1.0000  .6603  .2107  .2729  .1733  .0065 
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B 

.6603 

1.0000 

.1817 

-.0946 

-.2336 

-.4352 

z( 

1) 

.2107 

.1817 

1.0000 

.1293 

.0783 

.0060 

Z( 

2) 

.2729 

-.0946 

.1293 

1.0000 

.7988 

.5395 

z  ( 

3) 

.1733 

-.2336 

.0783 

.7988 

1.0000 

.6922 

Z( 

4) 

.0065 

-.4352 

.0060 

.5395 

.6922 

1.0000 

AREA  =  .7792  STD.  DEV.  (AREA)  =  .0536 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE- POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.1288 

(  .0401  , 

.3034 

) 

.010 

.1773 

(  .0686  , 

.3575 

) 

.020 

.2418 

(  .1139  , 

.4228 

) 

.030 

.2887 

(  .1508  , 

.4675 

) 

.040 

.3265 

(  .1825  , 

.5026 

) 

.050 

.3586 

(  .2105  , 

.5321 

) 

.060 

.3869 

(  .2357  , 

.5578 

) 

.070 

.4121 

(  .2586  , 

.5806 

) 

.080 

.4350 

(  .2796  , 

.6013 

) 

.090 

.4560 

(  .2991  , 

.6203 

) 

.100 

.4755 

(  .3172  , 

.6378 

) 

.110 

.4937 

(  .3342  , 

.6541 

) 

.120 

.5107 

(  .3502  , 

.6694 

) 

.130 

.5267 

(  .3653  , 

.6838 

) 

.140 

.5419 

(  .3796  , 

.6974 

) 

.150 

.5563 

(  .3932  , 

.7102 

) 

.200 

.6190 

(  .4527  , 

.7656 

) 

.250 

.6705 

(  .5018  , 

.8100 

) 

.300 

.7141 

(  .5440  , 

.8462 

) 

.400 

.7851 

(  .6147  , 

.9011 

) 

.500 

.8411 

(  .6741  , 

.9390 

) 

.600 

.8865 

(  .7271  , 

.9650 

) 

.700 

.9240 

(  .7771  , 

.9822 

) 

.800 

.9550 

(  .8270  , 

.9928 

) 

.900 

.9803 

(  .8821  , 

.9983 

) 

.950 

.9909 

(  .9165  , 

.9996 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(  .0170,  .2252)  (  .0054,  .1332) 

(  .0563,  .3768)  (  .0292,  .2853) 

(  .1015,  .4785)  (  .0621,  .3926) 

(  .9063,  .9817)  (  .8516,  .9687) 


UPPER 

BOUND 

( 

FPF  , 

TPF  ) 

( 

.0456, 

.3452) 

( 

.1002, 

.4759) 

( 

.1566, 

.5653) 

( 

.9444, 

.9897) 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  rion? 
(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  8/28/1998 

TIME  -  8:49:  3 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  06  Analog  FAD  resp  for  191  cases  available. 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  178. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  13. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .022  .101  .180  .826  1.000 

TPF:  .000  .231  .385  .615  .923  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .7746  B=  .7389 

Z(K)=  -.938  .916  1.275  2.006 

LOGL=  -211.5228 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .8337  B=  .7752 

Z(K)=  -.934  .904  1.299  2.003 

LOGL=  -211.4104 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

31.  115.  14.  14.  4. 

1.  4.  3.  2.  3. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1361 

.0446 

.0055 

.0060 

.0052 

.0019 

B 

.0446 

.0478 

.0026 

-.0007 

-.0031 

-.0118 

z( 

1) 

.0055 

.0026 

.0122 

.0027 

.0022 

.0013 

z( 

2) 

.0060 

-.0007 

.0027 

.0117 

.0098 

.0078 

z( 

3) 

.0052 

-  .0031 

.0022 

.0098 

.0162 

.0131 

z( 

4) 

.0019 

-.0118 

.0013 

.0078 

.0131 

.0404 

CORRELATION  MATRIX: 

A  1.0000  .5527  .1364  .1490  .1112  .0260 
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B 

.5527 

1.0000 

.1075 

-.0307 

-  .1108 

-.2691 

z( 

1) 

.1364 

.1075 

1.0000 

.2281 

.1549 

.0593 

z( 

2) 

.1490 

- .0307 

.2281 

1.0000 

.7102 

.3603 

z( 

3) 

.1112 

-.1108 

.1549 

.7102 

1.0000 

.5138 

z{ 

4) 

.0260 

-.2691 

.0593 

.3603 

.5138 

1.0000 

AREA  =  .7450  STD.  DEV. (AREA)  =  .0834 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.1224 

( 

.0183  , 

.4067 

) 

.010 

.1661 

( 

.0345  , 

.4518 

) 

.020 

.2241 

( 

.0631  , 

.5049 

) 

.030 

.2662 

( 

.0883  , 

.5408 

) 

.040 

.3003 

( 

.1111  , 

.5690 

) 

.050 

.3294 

( 

.1320  , 

.5925 

) 

.060 

.3551 

( 

.1513  , 

.6131 

) 

.070 

.3781 

( 

.1694  , 

.6314 

) 

.080 

.3991 

( 

.1864  , 

.6481 

) 

.090 

.4185 

( 

.2023  , 

.6634 

) 

,100 

.4365 

( 

.2175  , 

.6777 

) 

.110 

.4534 

( 

.2318  , 

.6910 

) 

.120 

.4693 

( 

.2455  , 

.7035 

) 

.130 

.4843 

{ 

.2586  , 

.7153 

) 

.140 

.4985 

( 

.2710  , 

.7265 

) 

.150 

.5121 

( 

.2830  , 

.7372 

) 

.200 

.5720 

( 

.3364  , 

.7839 

) 

.250 

.6221 

( 

.3815  , 

.8222 

) 

.300 

.6655 

{ 

.4207  , 

.8543 

) 

.400 

.7382 

( 

.4874  , 

.9044 

) 

.500 

.7978 

( 

.5440  , 

.9403 

) 

.600 

.8484 

( 

.5954  , 

.9655 

) 

.700 

.8925 

( 

.6451  , 

.9825 

) 

.800 

.9314 

( 

.6970  , 

.9930 

) 

.900 

.9662 

( 

.7590  , 

.9984 

) 

.950 

.9825 

( 

.8025  , 

.9996 

) 

ESTIMATES  OP  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

{  FPF  , 

TPF  ) 

{  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0226, 

.2360) 

(  .0083, 

.1528) 

(  .0538, 

.3395) 

(  .0970, 

.4312) 

(  .0607, 

.3569) 

(  .1470, 

.5081) 

(  .1830, 

.5529) 

(  .1322, 

.4874) 

(  .2446, 

.6170) 

(  .8248, 

.9403) 

{  .7635, 

.9177) 

(  .8749, 

.9577) 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 


(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name; 

Do  you  want  to  save  an  output  file  for  plotting  the 
(y/N,  or  R  to  restart) 


ROC  curve? 

DATE  -  8/28/1998 

TIME  -  8:49:  3 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  06  Digital  FAD  resp  for  191  cases  available. 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  178. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  13. 

RESPONSE  DATA: 

CATEGORY  1234 

ACTUALLY  NEGATIVE  CASES  17.  133.  20.  4. 

ACTUALLY  POSITIVE  CASES  0.  9.  3.  1. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .022  .045  .157  .904  1.000 

TPF;  .000  .000  .077  .308  1.000  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  .4203  B=  1.0617 

Z(K)=  -1.308  1.006  1.696  2.006 

LOGL=  -164.3522 

CHI-SQUARE  GOODNESS  OF  PIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.1942  B=  1.6646 

Z(K)=  -1.308  1.007  1.676  2.025 

LOGL=  -163.4719 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE-COVARIANCE  MATRIX : 


A 

.8199 

.5903 

.0069 

.0259 

-  .0107 

-  .0242 

B 

.5903 

.5219 

.0031 

.0052 

-.0278 

-.0389 

Z{ 

1) 

.0069 

.0031 

.0168 

.0021 

.0014 

.0012 

Z( 

2) 

.0259 

.0052 

.0021 

.0128 

.0096 

.0084 

z{ 

3) 

-.0107 

-.0278 

.0014 

.0096 

.0255 

.0231 

z( 

4) 

- .0242 

-.0389 

.0012 

.0084 

.0231 

.0436 

CORRELATION  MATRIX: 

A 

1.0000 

.9023 

.0588 

.2532 

-.0741 

-.1279 

5 

4  . 
0. 
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B 

.9023 

1.0000 

.0325 

.0632 

-  .2409 

-.2578 

z( 

1) 

.0588 

.0325 

1.0000 

.1434 

.0699 

.0446 

z{ 

2) 

.2532 

.0632 

.1434 

1.0000 

.5302 

.3549 

z{ 

3) 

-.0741 

-.2409 

.0699 

.5302 

1.0000 

.6925 

z{ 

4) 

-.1279 

-.2578 

.0446 

.3549 

.6925 

1.0000 

AREA  =  .7307  STD.  DEV. (AREA)  =  .0995 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

/ 

UPPER  BOUND) 

.005 

.0010 

( 

.0000 

/ 

.1815 

) 

.010 

.0037 

( 

.0000 

t 

.2058 

) 

.020 

.0130 

( 

.0001 

/ 

.2386 

) 

.030 

.0263 

( 

.0006 

# 

.2649 

) 

.040 

.0426 

( 

.0020 

/ 

.2890 

) 

.050 

.0612 

( 

.0047 

# 

.3126 

) 

.060 

.0816 

( 

.0090 

/ 

.3365 

) 

.070 

.1033 

( 

.0150 

/ 

.3612 

) 

.080 

.1261 

( 

.0226 

/ 

.3870 

) 

.090 

.1496 

( 

.0315 

/ 

.4140 

) 

.100 

.1737 

( 

.0415 

/ 

.4422 

) 

.110 

.1983 

( 

.0521 

/ 

.4717 

) 

.120 

.2230 

( 

.0631 

/ 

.5022 

) 

.130 

.2479 

( 

.0741 

t 

.5335 

) 

.140 

.2728 

( 

.0849 

1 

.5652 

) 

.150 

.2977 

( 

.0955 

§ 

.5970 

) 

.200 

.4182 

( 

.1413 

/ 

.7458 

) 

.250 

.5287 

( 

.1762 

/ 

.8585 

) 

.300 

.6262 

( 

.2036 

/ 

.9296 

) 

.400 

.7803 

( 

.2462 

/ 

.9872 

) 

.500 

.8838 

( 

.2808 

/ 

.9985 

) 

.600 

.9469 

( 

.3126 

/ 

.9999 

) 

.700 

.9806 

( 

.3451 

t 

1.0000 

) 

.800 

.9953 

'( 

.3818 

1.0000 

) 

.900 

.9996 

( 

.4313 

t 

1.0000 

) 

.950 

1.0000 

( 

.4714 

/ 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

(  FPF  ,  TPF  )  ( 

(  .0214,  .0147)  ( 

(  .0469,  .0553)  ( 

(  .1569,  .3147)  ( 

(  .9045,  .9996)  ( 


LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

.0075, 

.0021) 

(  .0530, 

.0673) 

.0234, 

.0172) 

(  .0865, 

.1413) 

.1095, 

.1972) 

(  .2161, 

,4549) 

.8539, 

.9984) 

(  .9409, 

.9999) 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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[eader06  Analog  ROC  Type=Diag  N=191  Area=.9008  STD  Area=.0325 


CO  lO  ^ 

d  o  d 

uojpeJd  3Aj;jS0cJ  ami 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  8/28/1998 

TIME  -  8:49:  3 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  06  Analog  Diag  resp  for  191  cases  available. 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  153. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  38. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  5.  110.  35.  3.  0. 

ACTUALLY  POSITIVE  CASES  0.  5.  16.  13.  4. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .000  .020  .248  .967  1.000 

TPF:  .000  .105  .447  .868  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.1385  B=  .7178 

Z(K)=  -1.843  .679  2.062  2.720 

LOGL=  -170.1710 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CTU^CULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.7216  B=  .8901 

Z(K)=  -1.845  .680  2.073  3.355 

LOGL=  -163.8364 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1357 

.0636 

.0037 

.0124 

-.0087 

-  .0882 

B 

.0636 

.0479 

.0015 

.0030 

-.0259 

-.0966 

z( 

1) 

.0037 

.0015 

.0387 

.0024 

.0009 

-  .0012 

Z( 

2) 

.0124 

.0030 

.0024 

.0121 

.0067 

.0026 

Z( 

3) 

-  .0087 

-  .0259 

.0009 

.0067 

.0530 

.0857 

z( 

4) 

-.0882 

-.0966 

-.0012 

.0026 

.0857 

.3101 

CORRELATION  MATRIX : 

A  1.0000  .7884  .0516  .3056  -.1022  -.4301 
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B 

.7884 

1.0000 

.0343 

.1253 

-.5131 

-.7930 

z( 

1) 

.0516 

.0343 

1.0000 

.1103 

.0194 

-.0108 

Z( 

2) 

.3056 

.1253 

.1103 

1.0000 

.2646 

.0421 

Z( 

3) 

-.1022 

-.5131 

.0194 

.2646 

1.0000 

.6682 

Z( 

4) 

-.4301 

-.7930 

- .0108 

.0421 

.6682 

1,0000 

AREA  =  .9008  STD.  DEV.  (AREA)  =  .0325 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.2839 

( 

.1025 

.5496 

) 

.010 

.3634 

( 

.1668 

.6057 

) 

.020 

.4575 

( 

.2580 

.6686 

) 

.030 

.5188 

( 

.3240 

.7092 

) 

.040 

.5648 

( 

.3756 

.7399 

) 

.050 

.6015 

( 

.4178 

.7648 

) 

.060 

.6321 

( 

.4533 

.7859 

) 

.070 

.6583 

( 

.4837 

.8042 

) 

.080 

.6811 

( 

.5101 

.8202 

) 

.090 

.7013 

( 

.5334 

.8346 

) 

.100 

.7193 

( 

.5541 

.8474 

) 

.110 

.7356 

( 

.5728 

.8591 

) 

.120 

.7504 

( 

.5897 

.8696 

) 

.130 

.7639 

( 

.6051 

.8793 

) 

.140 

.7764 

( 

.6192 

.8881 

) 

.150 

.7879 

( 

.6322 

.8963 

) 

.200 

.8346 

( 

.6851 

.9283 

) 

.250 

.8690 

( 

.7246 

.9502 

) 

.300 

.8953 

( 

.7559 

.9654 

) 

.400 

.9327 

( 

.8041 

.9837 

) 

.500 

.9574 

( 

.8412 

.9927 

) 

.600 

.9742 

( 

.8721 

.9971 

) 

.700 

.9857 

( 

.8994 

.9990 

) 

.800 

.9933 

( 

.9250 

.9998 

) 

,900 

.9979 

( 

.9514 

1.0000 

) 

.950 

.9993 

( 

.9668 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(  .0004,  .1031)  (  .0000,  .0127)  (  .0118,  .3848) 

(  .0191,  .4510)  (  .0058,  .2999)  (  .0525,  .6097) 

(  .2483,  .8679)  (  .1853,  .8224)  {  .3213,  .9046) 

(  .9674,  .9996)  (  .9277,  .9987)  (  .9871,  .9999) 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 


I _ 
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Reader06  Digital  ROC  Type=Diag  N=191  Area=.8165  STD  Area=.0504 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  r\ui? 
(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  8/28/1998 

TIME  -  8:49:  3 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  06  Digital  Diag  resp  for  191  cases  available. 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  153. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  38. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .007  .020  .183  .987  1.000 

TPF:  .000  .053  .289  .632  1.000  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .6959  B=  .7446 

Z(K)=  -2.225  .904  2.062  2.482 

LOGL=  -147.3942 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  6  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.2667  B=  .9860 

Z{K)=  -2.226  .910  1.946  2.794 

LOGL=  -142.4528 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

2.  123.  25.  2.  1. 

0.  14.  13.  9.  2. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1422 

.0751 

.0051 

.0167 

-  .0106 

-  .0567 

B 

.0751 

.0595 

.0025 

.0038 

-.0239 

-.0654 

z{ 

1) 

.0051 

.0025 

.0742 

.0020 

.0005 

-.0013 

z( 

2) 

.0167 

.0038 

.0020 

.0139 

.0092 

.0063 

z( 

3) 

-.0106 

-  .0239 

.0005 

.0092 

.0408 

.0525 

z{ 

4) 

-.0567 

-.0654 

-.0013 

,0063 

.0525 

.1552 

CORRELATION  MATRIX: 

A  1.0000  .8166  .0495  .3770  -.1392  -.3819 
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B 

.8166 

1.0000 

.0383 

.1315 

-  .4843 

-.6804 

z{ 

1) 

.0495 

.0383 

1.0000 

.0616 

.0087 

-.0120 

Z{ 

2) 

.3770 

.1315 

.0616 

1.0000 

.3857 

.1350 

z( 

3) 

-.1392 

-.4843 

.0087 

.3857 

1.0000 

.6601 

z( 

4) 

- .3819 

-.6804 

- .0120 

.1350 

.6601 

1.0000 

AREA  =  .8165  STD.  DEV.  (AREA)  =  .0504 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.1014 

( 

.0211  , 

.3035 

) 

.010 

.1521 

( 

.0454  , 

.3581 

) 

.020 

.2240 

( 

.0920  , 

.4253 

) 

.030 

.2782 

( 

.1341  , 

.4725 

) 

.040 

.3228 

( 

.1721  , 

.5106 

) 

.050 

.3611 

( 

.2062  , 

.5433 

) 

.060 

.3949 

( 

.2372  , 

.5723 

) 

.070 

.4252 

( 

.2653  , 

.5987 

) 

.080 

.4527 

( 

.2909  , 

.6229 

) 

.090 

.4779 

( 

.3143  , 

.6453 

) 

.100 

.5012 

( 

.3359  , 

.6663 

) 

.110 

.5228 

( 

.3558  , 

.6859 

) 

.120 

.5430 

( 

.3743  , 

.7043 

) 

.130 

.5620 

( 

.3915  , 

.7216 

) 

.140 

.5799 

( 

.4076  , 

.7379 

) 

.150 

.5967 

( 

.4226  , 

.7533 

) 

.200 

.6690 

( 

.4862  , 

.8182 

) 

.250 

.7264 

( 

.5363  , 

.8671 

) 

.300 

.7734 

( 

.5778  , 

.9038 

) 

.400 

.8455 

( 

.6455  , 

.9517 

) 

.500 

.8974 

( 

.7011  , 

.9776 

) 

.600 

.9353 

( 

.7503  , 

.9908 

) 

.700 

.9627 

( 

.7964  , 

.9969 

) 

.800 

.9820 

( 

.8423  , 

.9993 

) 

.900 

.9943 

( 

.8928  , 

.9999 

) 

.950 

.9981 

( 

.9243  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0026, 

.0684) 

(  .0002, 

.0123) 

(  .0216, 

.2338) 

(  .0258, 

.2572) 

(  .0096, 

.1486) 

(  .0606, 

.3968) 

(  .1814, 

.6441) 

(  .1270, 

.5564) 

(  .2485, 

.7247) 

(  .9870, 

.9997) 

(  .9547, 

.9983) 

(  .9971, 

1.0000) 
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Reader07  Analog  ROC  TypesMass  N=330  Areas.9409  STD  AreaB.0177 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/19/1998 
TIME  -  11:28:28 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  07  Analog  Mass  resp  for  330  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  280. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  50. 

RESPONSE  DATA: 

CATEGORY  123 

ACTUALLY  NEGATIVE  CASES  145.  91.  24. 

ACTUALLY  POSITIVE  CASES  1.  5.  3. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .043  .071  .157  .482  1.000 

TPF:  .000  .680  .820  .880  .980  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  2.1028  B=  .8971 

Z(K)=  .045  1.006  1.466  1.719 

LOGL=  -375.4041 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  2.1762  B=  .9692 

Z(K)=  .046  1.005  1.428  1.748 

LOGL=  -374.7759 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


4  5 

8.  12. 

7.  34. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1484 

.0807 

.0049 

.0063 

.0026 

-  .0043 

B 

.0807 

.0600 

.0017 

.0001 

-  .0047 

-  .0119 

z{ 

1) 

.0049 

.0017 

.0056 

.0031 

.0025 

.0021 

z{ 

2) 

.0063 

.0001 

.0031 

.0080 

.0067 

.0061 

z( 

3) 

.0026 

-.0047 

.0025 

.0067 

.0116 

.0111 

z( 

4) 

-.0043 

-.0119 

.0021 

.0061 

.0111 

.0176 

CORRELATION  MATRIX: 


A 

1.0000 

.8552 

.1700 

.1834 

.0634 

-  .0842 

B 

.8552 

1.0000 

.0909 

.0050 

-.1795 

-  .3679 

z( 

1) 

.1700 

.0909 

1.0000 

.4599 

.3080 

.2107 

z{ 

2) 

.1834 

.0050 

.4599 

1.0000 

.6966 

.5169 

z( 

3) 

.0634 

-.1795 

.3080 

.6966 

1.0000 

.7756 

z( 

4) 

-.0842 

-.3679 

.2107 

.5169 

.7756 

1.0000 
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AREA  =  .9409  STD.  DEV. (AREA)  =  .0177 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.3742 

( 

.1517 

# 

.6509 

) 

.010 

.4685 

( 

.2447 

t 

.7031 

) 

.020 

.5735 

( 

.3695 

t 

.7592 

) 

.030 

.6379 

( 

.4541 

f 

.7942 

) 

.040 

.6840 

( 

.5167 

f 

.  8202 

) 

.050 

.7196 

( 

.5654 

f 

.8410 

) 

.060 

.7482 

( 

.6045 

/ 

.8584 

) 

.070 

.7720 

( 

.6366 

f 

,8732 

) 

.080 

.7922 

( 

.6634 

t 

.8862 

) 

.090 

.8096 

( 

.6862 

/ 

.8976 

) 

.100 

.8248 

( 

.7059 

f 

.9076 

) 

.110 

.8382 

( 

.7230 

f 

.9166 

) 

.120 

.8502 

( 

.7381 

# 

.9246 

) 

.130 

.8609 

( 

.7515 

9 

.9318 

) 

.140 

.8706 

( 

.7635 

9 

.9383 

) 

.150 

.8793 

( 

.7743 

9 

.9441 

) 

.200 

.9132 

( 

.8161 

9 

.9657 

) 

.250 

.9361 

( 

.8452 

9 

.9788 

) 

.300 

.9524 

( 

.8671 

9 

.9869 

) 

.400 

.9733 

( 

.8992 

9 

.9951 

) 

.500 

.9852 

( 

.9224 

9 

.9983 

) 

.600 

.9923 

( 

.9406 

9 

.9995 

) 

.700 

.9964 

( 

.9559 

9 

.9999 

) 

.800 

.9986 

( 

.9694 

9 

1.0000 

) 

.900 

.9997 

( 

.9821 

9 

1.0000 

) 

.950 

.9999 

( 

.9888 

9 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOXJNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0402, 

.6850) 

(  .0223, 

.5909) 

(  .0683, 

.7684) 

(  .0766, 

.7858) 

(  ,0506, 

.7215) 

(  ,1118, 

.8405) 

(  .1574, 

.8853) 

(  .1190, 

.8491) 

(  .2032, 

.9149) 

(  .4816, 

.9835) 

(  .4236, 

.9767) 

(  .5401, 

.9885) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  r\an? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(y/N,  or  Q  to  quit) 
enter  INPUT  file  name; 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(y/N,  or  R  to  restart) 

DATE  -  11/19/1998 
TIME  -  11:28:28 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  07  Analog  calc  resp  for  330  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  253. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  77. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  150.  61.  8.  16.  18. 

ACTUALLY  POSITIVE  CASES  4.  5.  1.  7.  60. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .071  .134  .166  .407  1.000 

TPF:  .000  .779  .870  .883  .948  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.8127  B=  .6715 

Z(K)=  .235  .970  1.106  1.468 

LOGL=  -346.7852 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.8400  B=  .7189 

Z(K)=  .237  .960  1.088  1.481 

LOGL=  -346.5920 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0769 

.0426 

.0048 

.0038 

.0031 

-  .0005 

B 

.0426 

.0372 

.0018 

-.0009 

-  .0020 

-.0071 

z( 

1) 

.0048 

.0018 

.0063 

.0041 

.0038 

.0031 

z( 

2) 

.0038 

-  .0009 

.0041 

.0085 

.0080 

.0071 

z{ 

3) 

.0031 

-  .0020 

.0038 

.0080 

.0093 

.0083 

z( 

4) 

-.0005 

-  .0071 

.0031 

.0071 

.0083 

.0141 

CORRELATION  MATRIX: 


A 

1.0000 

.7955 

.2179 

.1494 

.1176 

-.0166 

B 

.7955 

1.0000 

.1140 

-.0526 

-.1065 

-.3107 

Z{ 

1) 

.2179 

.1140 

1.0000 

.5593 

.4983 

.3289 

z( 

2) 

.1494 

-.0526 

.5593 

1.0000 

.9031 

.6481 

z( 

3) 

.1176 

-.1065 

.  .4983 

.9031 

1.0000 

.7268 

z{ 

4) 

-.0166 

-.3107 

.3289 

.6481 

.7268 

1.0000 
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AREA  =  .9324  STD.  DEV.  (AREA)  =  .0187 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

/ 

UPPER  BOUND) 

.005 

.4952 

( 

.2590 

/ 

.7331 

) 

.010 

.5664 

( 

.3488 

f 

.7652 

) 

.020 

.6418 

( 

.4548 

/ 

.7995 

) 

.030 

.6871 

( 

.5222 

/ 

.8210 

) 

.040 

.7194 

( 

.5712 

/ 

.8372 

) 

.050 

.7445 

( 

.6091 

/ 

.8502 

) 

.060 

.7648 

( 

.6397 

/ 

.8613 

) 

.070 

.7819 

( 

.6650 

# 

.8711 

) 

.080 

.7966 

( 

.6864 

/ 

.8797 

) 

.090 

.8095 

( 

.7047 

f 

.8876 

) 

.100 

.8208 

( 

.7207 

/ 

.8947 

) 

.110 

.8310 

( 

.7347 

t 

.9013 

) 

.120 

.8402 

( 

.7471 

t 

.9074 

) 

.130 

.8485 

( 

.7582 

f 

.9130 

) 

.140 

.8562 

( 

.7682 

/ 

.9183 

) 

.150 

.8632 

( 

.7773 

/ 

.9232 

) 

.200 

.8916 

( 

.8123 

! 

.9434 

) 

.250 

.9123 

( 

.8367 

/ 

.9581 

) 

.300 

.9283 

( 

.8552 

/ 

.9691 

) 

.400 

.9514 

( 

.8824 

t 

.9834 

) 

.500 

.9671 

( 

.9026 

t 

.9914 

) 

.600 

.9784 

( 

.9191 

/ 

.9959 

) 

.700 

.9867 

( 

.9339 

/ 

.9983 

) 

.800 

.9928 

( 

.9479 

/ 

.9995 

) 

.900 

.9971 

( 

.9630 

/ 

.9999 

) 

.950 

.9987 

( 

.9724 

# 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0692, 

.7808) 

(  .0433, 

.7284) 

(  .1058, 

.8269) 

(  .1383, 

.8549) 

(  .1008, 

.8217) 

(  .1843, 

.8837) 

(  .1685, 

.8749) 

(  .1271, 

.8462) 

(  .2177, 

.8996) 

(  .4064, 

.9525) 

(  .3473, 

.9403) 

(  .4678, 

.9626) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC 
(Y/N,  or  R  to  restart) 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


file  for  the  next  run? 


curve? 

DATE  -  11/19/1998 
TIME  -  11:28:28 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  R  O  C  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  07  Analog  FAD  resp  for  330  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  308. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  22. 

RESPONSE  DATA: 

5 

19, 

6. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .062  .133  .166  .422  1.000 

TPF:  .000  .273  .545  .591  .818  1.000 


CATEGORY  1234 
ACTUALLY  NEGATIVE  CASES  178.  79.  10.  22, 
ACTUALLY  POSITIVE  CASES  4.  5.  1.  6, 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.1957  B=  1.0823 

Z(K)=  .196  .972  1.112  1.541 

LOGL=  -3  83.9,933 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.2041  B=  1.1216 

Z{K)=  .199  .960  1.095  1.558 

LOGL=  -383.8640 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1283 

.0741 

.0061 

.0044 

.0036 

-.0003 

B 

.0741 

.0794 

.0023 

-.0017 

-.0029 

-.0087 

z( 

1) 

.0061 

.0023 

.0052 

.0032 

.0030 

.0024 

Z( 

2) 

.0044 

-.0017 

.0032 

.0070 

.0066 

.0056 

z( 

3) 

.0036 

-.0029 

.0030 

.0066 

.0078 

.0066 

z( 

4) 

- .0003 

-.0087 

.0024 

.0056 

.0066 

.0127 

CORRELATION  MATRIX: 

A 

1.0000 

.7339 

.2361 

.1469 

.1150 

-  .0071 

B 

.7339 

1.0000 

.1117 

-.0711 

-.1176 

- .2752 

z( 

1) 

.2361 

.1117 

1-0000 

.5400 

.4770 

.2942 

Z{ 

2) 

.1469 

-.0711 

.5400 

1.0000 

.8937 

.5944 

z( 

3) 

.1150 

-.1176 

.4770 

.8937 

1.0000 

.6695 

z( 

4) 

- .0071 

- .2752 

.2942 

.5944 

.6695 

1.0000 
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AREA  =  .7885  STD.  DEV. (AREA)  =  .0503 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.0460 

( 

.010 

.0799 

( 

.020 

.1357 

( 

.030 

.1825 

( 

.040 

.2237 

( 

.050 

.2607 

( 

.060 

.2946 

( 

.070 

.3258 

( 

.080 

.3549 

( 

.090 

.3821 

( 

.100 

.4077 

( 

.110 

.4318 

( 

.120 

.4547 

( 

.130 

.4764 

( 

.140 

.4970 

( 

.150 

.5166 

( 

.200 

.6027 

( 

.250 

.6729 

( 

.300 

.7312 

( 

.400 

.8213 

( 

.500 

.8857 

( 

.600 

.9316 

( 

.700 

.9634 

( 

.800 

.9841 

( 

.900 

.9959 

( 

.950 

.9989 

( 

.0034  , 

.2546 

) 

.0104  , 

.3086 

) 

.0299  , 

.3752 

) 

.0532  , 

.4217 

) 

.0783  , 

.4590 

) 

.1039  , 

.4909 

) 

.1297  , 

.5190 

) 

.1551  , 

.5446 

) 

.1799  , 

.5681 

) 

.2040  , 

.5900 

) 

.2273  , 

.6106 

) 

.2497  , 

.6301 

) 

.2712  , 

.6485 

) 

.2918  , 

.6662 

) 

.3116  , 

.6830 

) 

.3305  , 

.6991 

) 

.4133  , 

.7703 

) 

.4801  , 

.8279 

) 

.5354  , 

.8737 

) 

.6230  , 

.9367 

) 

.6922  , 

.9717 

) 

.7510  , 

.9892 

) 

.8041  , 

.9968 

) 

.8550  , 

.9994 

’ ) 

.9080  , 

1.0000 

) 

.9391  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(  .0597,  .2936)  (  .0377,  .2146)  (  .0906,  .3839) 

(  .1367,  .4903)  (  .1024,  .4137)  (  .1781,  .5672) 

(  .1685,  .5506)  (  .1305,  .4775)  (  .2130,  .6221) 

(  .4213,  .8368)  (  .3671,  .7948)  (  .4770,  .8727) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/19/1998 
TIME  -  11:28:28 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  07  Analog  Diag  resp  for  330  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  240. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  90. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  51.  120.  59.  10.  0. 

ACTUALLY  POSITIVE  CASES  1.  10.  53.  14.  12. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .000  .042  .287  .788  1.000 

TPF:  .000  .133  .289  .878  .989  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  1.5147  B=  .9804 

Z(K)=  -.798  .560  1.732  2.866 

LOGL=  -391.6062 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.6642  B=  1.1390 

Z(K)=  -.791  .537  1.865  2.495 

LOGL=  -385.7740 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0492 

.0235 

.0045 

.0080 

.0006 

-.0088 

B 

.0235 

.0233 

.0017 

.0015 

-  .0124 

-  .0237 

z{ 

1) 

.0045 

.0017 

.0082 

.0028 

.0012 

.0004 

z( 

2) 

.0080 

.0015 

.0028 

.0070 

.0043 

.0035 

z{ 

3) 

.0006 

-.0124 

.0012 

.0043 

.0225 

.0267 

z( 

4) 

-.0088 

-.0237 

.0004 

.0035 

.0267 

.0504 

CORRELATION  MATRIX: 

A 

1.0000 

.6941 

.2230 

.4300 

.0188 

-.1759 

B 

.6941 

1.0000 

.1239 

,1187 

-  .5418 

-  .6905 

Z{ 

1) 

.2230 

.1239 

1.0000 

.3630 

.0861 

.0173 

z( 

2) 

.4300 

.1187 

.3630 

1.0000 

.3440 

.1880 

z( 

3) 

.0188 

-.5418 

.0861 

.3440 

1.0000 

.7930 

z( 

4) 

-.1759 

-.6905 

.0173 

.1880 

.7930 

1.0000 
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AREA  =  .8639  STD.  DEV. (AREA)  =  .0234 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION; 


FPF 

TPF 

(LOWER  BOUND 

/ 

UPPER  BOUND) 

.005 

.1020 

( 

.0333 

.2401 

) 

.010 

.1621 

( 

.0682 

.3148 

) 

.020 

.2497 

( 

.1318 

.4079 

) 

.030 

.3162 

( 

.1878 

.4718 

) 

.040 

.3706 

( 

.2376 

.5214 

) 

.050 

.4170 

( 

.2823 

.5623 

) 

.060 

.4574 

( 

.3227 

.5972 

) 

.070 

.4932 

( 

.3595 

.6277 

) 

.080 

.5253 

( 

.3931 

.6548 

) 

.090 

.5544 

( 

.4240 

.6791 

) 

.100 

.5809 

( 

.4526 

.7012 

) 

.110 

.6053 

( 

.4790 

.7213 

) 

.120 

.6277 

( 

.5035 

.7398 

) 

.130 

.6485 

( 

.5264 

.7568 

) 

.140 

.6677 

( 

.5477 

.7726 

) 

.150 

.6857 

( 

.5677 

.7872 

) 

.200 

.7598 

( 

.6513 

.8468 

) 

.250 

.8150 

( 

.7149 

.8897 

) 

.300 

.8571 

( 

.7650 

.9211 

) 

.400 

.9156 

( 

.8389 

.9610 

) 

.500 

.9520 

( 

.8905 

.9821 

) 

.600 

.9745 

( 

.9281 

,9927 

) 

.700 

.9881 

( 

.9560 

.9976 

) 

.800 

.9956 

( 

,9765 

.9994 

) 

.900 

.9991 

( 

.9911 

.9999 

) 

.950 

.9998 

( 

,9964 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  {  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

{  .0063,  .1195)  (  .0017,  .0466)  {  .0199,  .2494) 

{  .0311,  .3227)  (  .0154,  .2133)  (  .0581,  .4502) 

(  .2956,  .8537)  (  .2415,  .8066)  (  .3547,  .8925) 

(  .7855,  .9948)  {  .7303,  .9909)  (  .8335,  .9972) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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ReaderOS  Analog  ROC  Type=mass  N^SSO  Areas.1.0162  STD  Area=.5309 
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R  O  C  F  I  T  (IBM  VERSION  1.2) 


DATE  -  11/20/1998 
TIME  -  9:19:49 


MAXIMUM 
OF  A 
F 


L  I  K  E  L  I 
B  I  N  O  R  M 
ROM  RAT 


HOOD  E  S  T 
A  L  ROC  C 
I  N  G  DATA 


I  M  A  T  I  O  N 
U  R  V  E 


NAME  OF  INPUT  FILE  BEING  USED:  adiag08.txt 


DATA  DESCRIPTION;  Reader  08  Analog  diag  resp  for  330  cases  available 


OBSERVED  OPERATING  POINTS: 

FPF:  .000  .000  .008  .075  .275  1.000 

TPF;  .000  .222  .378  .611  .756  1.000 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.0162  B=  .5309 

CORRESPONDING  AREA  INDEX  (A-SUB-Z)  =  .8153  WITH  STD.  DEV.  =  .0376. 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.3626 

( 

.010 

.4132 

( 

.020 

.4703 

( 

.030 

.5069 

( 

.040 

.5345 

( 

.050 

.5567 

( 

.060 

.5756 

( 

.070 

.5919 

( 

.080 

.6064 

( 

.090 

,6195 

( 

.100 

.6314 

( 

.110 

.6424 

( 

.120 

.6526 

( 

.130 

.6621 

( 

.140 

-6710 

( 

.150 

.6794 

( 

.200 

.7155 

( 

.250 

.7448 

( 

.300 

,7697 

( 

.400 

.8111 

( 

.500 

.8452 

( 

.600 

.8750 

( 

.700 

.9022 

( 

.800 

.9283 

( 

.900 

.9551 

( 

.950 

.9706 

( 

2364  , 

.5059 

) 

2907  , 

.5450 

) 

3540  , 

.5893 

) 

3950  , 

.6183 

) 

4258  , 

.6406 

) 

4505  , 

.6590 

) 

4712  , 

.6748 

) 

4890  , 

.6888 

) 

5047  , 

.7014 

) 

5186  , 

.7128 

) 

5312  , 

.7234 

) 

5427  , 

.7332 

) 

5532  , 

.7424 

) 

5630  , 

.7510 

) 

5720  , 

.7592 

) 

5805  , 

.7669 

) 

6164  , 

.8004 

) 

6448  , 

.8277 

) 

6687  , 

.8507 

) 

7082  , 

.8879 

) 

7413  , 

.9170 

) 

7714  , 

.9403 

) 

8004  , 

.9596 

) 

8308  , 

,9755 

) 

8669  , 

.9887 

) 

8921  , 

.9945 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
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CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS : 


EXPECTED  OPERATING  POINT 


(  FPF  ,  TPF  )  { 

(  .0004,  .2222)  ( 

(  .0067,  .3830)  ( 

(  .0772,  .6026)  { 

(  .2746,  .7575)  { 


LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

.0000, 

.1132) 

(  .0059, 

.3746) 

.0016, 

.2899) 

(  .0232, 

.4834) 

.0498, 

.5564) 

(  .1148, 

.6474) 

.2212, 

.7285) 

(  .3337, 

.7846) 
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Reader  08  Analog  ROC  Type=calc  N=330  Area=.7387  STD  Area=.0846 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name; 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/20/1998 
TIME  -  9:11:57 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  08  Analog  calc  resp  for  330  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  253. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  77. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .032  .047  .055  .083  1.000 

TPF;  .000  .519  .584  .584  .584  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.5166  B=  .7803 

Z(K)=  1.385  1.596  1.671  1.858 

LOGL=  -174.3543 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .6729  B=  .3286 

Z(K)=  1.387  1.549  1.604  1.890 

LOGL=  -170.6075 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

232.  7.  2.  4.  8. 

32.  0.  0.  5.  40. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0915 

.0440 

.0065 

.0023 

.0007 

-.0098 

B 

.0440 

.0275 

.0016 

-  .0012 

-.0023 

-  .0095 

Z( 

1) 

.0065 

.0016 

.0129 

.0120 

.0118 

.0104 

z( 

2) 

.0023 

-.0012 

.0120 

.0150 

.0147 

.0139 

z( 

3) 

.0007 

-.0023 

.0118 

.0147 

.0159 

.0153 

z  ( 

4) 

-.0098 

-.0095 

.0104 

.0139 

.0153 

.0250 

CORRELATION  MATRIX: 

A 

1.0000 

.8787 

.1884 

.0626 

.0189 

- .2049 

B 

.8787 

1.0000 

.0856 

-  .0598 

-.1098 

- .3633 

Z( 

1) 

.1884 

.0856 

1.0000 

.8662 

.8199 

.5815 

Z( 

2) 

.0626 

-.0598 

.8662 

1.0000 

.9545 

.7200 

z{ 

3) 

.0189 

-.1098 

.8199 

.9545 

1.0000 

.7670 

z( 

4) 

-.2049 

-.3633 

.5815 

.7200 

.7670 

1.0000 
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AREA  =  .7387  STD.  DEV. (AREA)  =  .0846 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

UPPER  BOUND) 

.005 

.4310 

( 

.2750 

# 

.5989 

) 

.010 

.4635 

( 

.3229 

t 

.6087 

) 

.020 

.4991 

( 

.3742 

9 

.6242 

) 

.030 

.5218 

( 

.4042 

9 

.6375 

) 

.040 

.5388 

( 

.4248 

9 

.6497 

) 

.050 

.5526 

( 

.4399 

9 

.6611 

) 

.060 

.5643 

( 

.4516 

9 

.6720 

) 

.070 

.5745 

( 

.4609 

9 

.6822 

) 

.080 

.5836 

( 

.4684 

9 

.6919 

) 

.090 

.5918 

( 

.4747 

9 

.7012 

) 

.100 

.5994 

( 

.4800 

9 

.7100 

) 

.110 

.6063 

( 

.4846 

9 

.7184 

) 

.120 

.6128 

( 

.4886 

9 

.7265 

) 

.130 

.6189 

{ 

.4920 

9 

.7341 

) 

.140 

.6247 

( 

.4951 

9 

.7415 

) 

.150 

.6302 

{ 

.4978 

9 

.7486 

) 

.200 

.6541 

( 

.5081 

9 

.7801 

) 

.250 

.6741 

{ 

.5150 

9 

.8065 

) 

.300 

.6917 

( 

.5199 

9 

.8293 

) 

.400 

.7223 

( 

.5270 

9 

.8669 

) 

.500 

.7495 

( 

.5319 

9 

.8972 

) 

.600 

.7752 

( 

.5358 

9 

.9225 

) 

.700 

.8010 

( 

.5391 

9 

.9443 

) 

.800 

.8288 

( 

.5423 

9 

.9635 

) 

.900 

.8630 

( 

.5458 

9 

.9809 

) 

.950 

.8875 

( 

.5481 

9 

-9894 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

{  .0294, 

.5207) 

(  .0139, 

.4801) 

(  .0571, 

.5611) 

(  .0544, 

.5580) 

(  .0321, 

.5257) 

(  .0875, 

.5898) 

(  .0607, 

.5651) 

(  .0368, 

.5339) 

(  .0952, 

.5958) 

(  .0827, 

.5859) 

(  .0537, 

.5572) 

(  .1221, 

.6142) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/20/1998 
TIME  -  9:11:57 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  08  Analog  FAD  resp  for  330  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  308. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  22. 

RESPONSE  DATA: 

CATEGORY  1  2  34  5 

ACTUALLY  NEGATIVE  CASES  250.  24.  11.  14.  9. 

ACTUALLY  POSITIVE  CASES  11.  1.  3.  4.  3. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .029  .075  .110  .188  1.000 

TPF:  .000  .136  .318  .455  .500  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.1199  B=  1.1322 

Z(K)=  .884  1.225  1.442  1.893 

LOGL=  -256.1230 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .9484  B=  1.0304 

Z(K)=  .886  1.200  1.436  1.914 

LOGL=  -255.8990 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.2375 

.1413 

.0093 

.0052 

.0012 

-.0103 

B 

.1413 

.1155 

.0027 

-  .0013 

-.0052 

-.0159 

Z(  1) 

.0093 

.0027 

.0068 

.0058 

.0052 

,0042 

Z(  2) 

.0052 

-.0013 

.0058 

.0085 

.0077 

.0067 

Z(  3) 

.0012 

-.0052 

.0052 

.0077 

.0108 

.0096 

Z(  4) 

-.0103 

-.0159 

.0042 

.0067 

.0096 

.0210 

CORRELATION  MATRIX: 


A 

1.0000 

.8531 

.2320 

.1168 

.0237 

-.1460 

B 

.8531 

1.0000 

.0946 

-.0420 

-  .1462 

- .3223 

z( 

1) 

.2320 

.0946 

1.0000 

.7650 

.6084 

.3533 

z( 

2) 

.1168 

-.0420 

.7650 

1.0000 

.8094 

.5022 

z( 

3) 

.0237 

-.1462 

.6084 

.8094 

1.0000 

.6365 

z( 

4) 

-.1460 

- .3223 

.3533 

.5022 

.6365 

1.0000 
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AREA  =  .7455  STD.  DEV. (AREA)  =  .0804 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.0440 

( 

.0031 

f 

.2495 

) 

.010 

.0737 

( 

.0097 

/ 

.2875 

) 

.020 

.1214 

( 

.0279 

t 

.3361 

) 

.030 

.1611 

( 

.0491 

t 

.3721 

) 

.040 

.1960 

( 

.0713 

t 

.4029 

) 

.050 

.2276 

( 

.0936 

/ 

.4307 

) 

.060 

.2566 

( 

.1152 

/ 

.4567 

) 

.070 

.2835 

( 

.1359 

/ 

.4815 

) 

.080 

.3087 

( 

.1556 

/ 

.5053 

) 

.090 

.3324 

( 

.1741 

t 

.5284 

) 

.100 

.3548 

( 

.1915 

f 

.5508 

) 

.110 

.3762 

( 

.2078 

/ 

.5726 

) 

.120 

.3965 

( 

.2231 

/ 

.5937 

) 

.130 

.4159 

( 

.2373 

/ 

.6142 

) 

.140 

.4345 

( 

.2507 

/ 

.6341 

) 

.150 

.4524 

( 

.2633 

t 

.6533 

) 

.200 

.5324 

( 

.3162 

/ 

.7392 

) 

.250 

.6001 

( 

.3577 

r 

.8084 

) 

.300 

.6585 

( 

.3921 

/ 

.8623 

) 

.400 

.7542 

( 

.4489 

/ 

.9337 

) 

.500 

.8285 

( 

.4973 

f 

.9715 

) 

.600 

.8867 

( 

.5424 

/ 

.9896 

) 

.700 

.9317 

( 

.5879 

/ 

.9971 

) 

.800 

.9653 

( 

.6377 

# 

.9995 

) 

.900 

.9884 

( 

.7010 

/ 

1.0000 

) 

.950 

.9959 

( 

.7481 

/ 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0278, 

.1530) 

(  .0140, 

.0940) 

(  .0515, 

.2323) 

(  .0756, 

.2978) 

(  .0506, 

.2295) 

(  .1090, 

.3741) 

(  .1152, 

.3868) 

(  .0838, 

.3178) 

(  .1541, 

.4596) 

(  .1878, 

.5141) 

(  .1473, 

.4477) 

(  .2345, 

.5801) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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1 


R  O  C  F  I  T  (IBM  VERSION  1.2) 


DATE  -  11/20/1998 
TIME  -  9:21:37 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


NAME  OF  INPUT  FILE  BEING  USED:  adiag08.txt 


DATA  DESCRIPTION;  Reader  08  Analog  diag  resp  for  330  cases  available 


OBSERVED  OPERATING  POINTS: 

FPF:  .000  .000  .008  .075  .275  1.000 

TPF:  .000  .222  .378  .611  .756  1.000 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.0162  B=  .5309 

CORRESPONDING  AREA  INDEX  (A-SUB-Z)  =  .8153  WITH  STD.  DEV.  =  .0376. 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.3626 

( 

.2364  , 

.5059 

) 

.010 

.4132 

( 

.2907  , 

.5450 

) 

.020 

.4703 

( 

.3540  , 

-5893 

) 

.030 

.5069 

( 

.3950  , 

.6183 

) 

.040 

.5345 

( 

.4258  , 

.6406 

) 

.050 

.5567 

( 

.4505  , 

.6590 

) 

.060 

.5756 

( 

.4712  , 

.6748 

) 

.070 

.5919 

( 

.4890  , 

.6888 

) 

.080 

.6064 

( 

.5047  , 

.7014 

) 

.090 

.6195 

( 

.5186  , 

.7128 

) 

.100 

.6314 

( 

.5312  , 

.7234 

) 

.110 

.6424 

( 

.5427  , 

.7332 

) 

.120 

.6526 

( 

.5532  , 

.7424 

) 

.130 

.6621 

( 

.5630  , 

.7510 

) 

.140 

.6710 

( 

.5720  , 

.7592 

) 

.150 

.6794 

( 

.5805  , 

.7669 

) 

.200 

.7155 

( 

.6164  , 

.8004 

) 

.250 

.7448 

( 

.6448  , 

.8277 

) 

.300 

.7697 

( 

.6687  , 

.8507 

) 

.400 

.8111 

( 

.7082  , 

.8879 

) 

.500 

.8452 

( 

.7413  , 

.9170 

) 

.600 

.8750 

( 

.7714  , 

.9403 

) 

.700 

.9022 

( 

.8004  , 

.9596 

) 

.800 

.9283 

( 

.8308  , 

.9755 

) 

.900 

.9551 

( 

.8669  , 

.9887 

) 

.950 

.9706 

( 

.8921  , 

.9945 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
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CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 


{ 

FPF  , 

TPF  ) 

( 

( 

.0004, 

.2222) 

( 

( 

.0067, 

.3830) 

( 

{ 

.0772, 

.6026) 

( 

( 

-2746, 

.7575) 

( 

LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

.0000, 

.1132) 

(  .0059, 

.3746) 

.0016, 

.2899) 

{  .0232, 

.4834) 

.0498, 

.5564) 

(  .1148, 

.6474) 

.2212, 

.7285) 

(  .3337, 

.7846) 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart)  ,  , 

DATE  -  11/20/1998 

TIME  -  12:25:29 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  09  Analog  Mass  resp  for  330  cases 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  280. 

NO;  OF  ACTUALLY  POSITIVE  CASES  =  50. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  182.  61.  27.  5.  5. 

ACTUALLY  POSITIVE  CASES  6.  6.  9.  7.  22. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .018  .036  .132  .350  1.000 

TPF:  .000  .440  .580  .760  .880  1.000 

INITIAL  VALUES  OP  PARAMETERS: 

A=  1.5019  B=  .7541 

Z(K)=  .385  1.116  1.803  2.101 

LOGL=  -348.2317 
GOODNESS  OF  FIT  -- 

CHI  SQUARE=  1.7330  WITH  2  DEGREES  OF  FREEDOM,  P=  .4204 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.5153  B=  .7636 

Z{K)=  .388  1.106  1.783  2.158 

LOGL=  -347.8726 
GOODNESS  OF  FIT  -- 

CHI  SQUARE=  .8153  WITH  2  DEGREES  OF  FREEDOM,  P=  .6652 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0766 

.0343 

.0049 

.0041 

-.0007 

-.0072 

B 

.0343 

.0255 

.0015 

-  .0006 

-  .0069 

-.0138 

z( 

1) 

.0049 

.0015 

.0059 

.0040 

.0029 

.0023 

z{ 

2) 

.0041 

-.0006 

.0040 

.0085 

.0068 

.0065 

z{ 

3) 

-.0007 

-.0069 

.0029 

.0068 

.0178 

.0181 

Z( 

4) 

-.0072 

-  .0138 

.0023 

.0065 

.0181 

.0313 

CORRELATION  MATRIX: 

A 

1.0000 

.7766 

.2295 

.1603 

-  .0196 

-  .1471 

B 

.7766 

1.0000 

.1196 

-.0391 

-.3221 

-.4868 

z( 

1) 

.2295 

.1196 

1.0000 

.5596 

.2796 

.1708 

z( 

2) 

.1603 

-.0391 

.5596 

1.0000 

.5576 

.4019 

z{ 

3) 

- .0196 

-.3221 

.2796 

.5576 

1.0000 

.7682 

z( 

4) 

-.1471 

-.4868 

.1708 

.4019 

.7682 

1.0000 
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AREA  = 


.8858 


STD.  DEV. (AREA) 


.0307 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.3256 

( 

.010 

.3968 

{ 

.020 

.4787 

{ 

.030 

.5314 

( 

.040 

.5707 

( 

.050 

.6022 

( 

.060 

.6284 

( 

.070 

.6510 

( 

.080 

.6708 

( 

.090 

.6884 

( 

.100 

.7042 

( 

.110 

.7185 

( 

.120 

.7317 

( 

.130 

,7438 

( 

.140 

.7550 

( 

.150 

.7654 

( 

.200 

.8086 

( 

.250 

.8414 

( 

.300 

.8676 

{ 

.400 

.9069 

( 

.500 

.9351 

{ 

.600 

.9562 

( 

.700 

.9723 

( 

.800 

.9845 

( 

.900 

.9937 

{ 

.950 

.9972 

( 

.1667  , 

.5252 

) 

.2354  , 

.5785 

) 

.3224  , 

.6384 

) 

.3813  , 

.6771 

) 

.4260  , 

.7063 

) 

.4619  , 

.7302 

) 

.4919  , 

.7504 

) 

.5175  , 

.7680 

) 

.5397  , 

.7836 

) 

.5594  , 

.7976 

) 

.5769  , 

.8103 

) 

.5928  , 

.8218 

) 

.6072  , 

.8324 

) 

.6204  , 

.8422 

) 

.6325  , 

.8513 

) 

.6438  , 

.8597 

) 

.6901  , 

.8942 

) 

.7255  , 

.9195 

) 

.7541  , 

.9386 

) 

.7991  , 

.9645 

) 

.8347  , 

.9802 

) 

.8648  , 

.9897 

) 

.8921  , 

.9952 

) 

.9181  , 

.9983 

) 

.9455  , 

.9996 

) 

.9620  , 

.9999 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0155, 

.4472) 

(  .0061, 

.3455) 

(  .0350, 

.5525) 

(  .0373, 

.5612) 

(  .0205, 

.4818) 

(  .0641, 

.6382) 

{  .1344, 

.7488) 

(  .0992, 

.7030) 

(  .1773, 

.7905) 

(  .3492, 

.8886) 

(  .2952, 

.8653) 

{  .4064, 

.9090) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC 
(Y/N,  or  R  to  restart) 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


file  for  the  next  run? 


curve? 

DATE  -  11/20/1998 
TIME  -  12:25:29 


MAX 


IMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DA  T  A 


DATA  DESCRIPTION:  Reader  09  Analog  Calc  resp  for  330  cases 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  253. 
NO.  OF  ACTUALLY  POSITIVE  CASES  =  77. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  234.  9. 

CASES  23.  1. 


3  4  5 

0.  2.  8. 

2.  4.  47. 


OBSERVED  OPERATING  POINTS: 

FPF:  .000  .032  .040  .040  .075  1.000 

TPF:  .000  .610  .662  .688  .701  1.000 


INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3470  B=  .5471 

Z{K)=  1.439  1.657  1.757  1.858 

LOGL=  -165.2280 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 


FINAL  VALUES  OF  PARAMETERS: 

A=  1.3454  B=  .5618 

Z(K)=  1.441  1.669  1.723  1.889 

LOGL=  -164.0483 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1996 

.1072 

.0119 

- .0013 

-  .0050 

-.0182 

B 

.1072 

.0656 

.0029 

-  .0059 

-  .0083 

-  .0170 

z( 

1) 

.0119 

.0029 

.0137 

.0124 

.0122 

.0114 

z( 

2) 

-.0013 

-.0059 

.0124 

.0171 

.0171 

.0172 

z( 

3) 

-.0050 

-.0083 

.0122 

.0171 

.0185 

.0189 

Z( 

4) 

- .0182 

-.0170 

.0114 

.0172 

.0189 

.0249 

CORRELATION  MATRIX: 

A 

1.0000 

.9365 

.2280 

-  .0227 

-.0827 

-.2584 

B 

.9365 

1.0000 

.0979 

-.1750 

-  .2386 

- .4213 

z( 

1) 

.2280 

.0979 

1.0000 

.8133 

.7652 

.6151 

z( 

2) 

-.0227 

-.1750 

.8133 

1.0000 

.9610 

.8355 

z( 

3) 

-.0827 

-.2386 

.7652 

.9610 

1.0000 

.8812 

Z{ 

4) 

-.2584 

-.4213 

.6151 

.8355 

.8812 

1.0000 
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AREA  =  .8796  STD.  DEV.  (AREA)  =  .0548 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.4594 

( 

.2526 

/ 

.6781 

) 

.010 

.5152 

( 

.3351 

f 

.6923 

) 

.020 

.5759 

( 

.4280 

f 

.7136 

) 

.030 

.6135 

( 

.4830 

f 

.7323 

) 

.040 

.6412 

( 

.5195 

/ 

.7499 

) 

.050 

.6632 

( 

.5454 

/ 

.7668 

) 

.060 

.6814 

( 

.5644 

t 

.7827 

) 

.070 

.6971 

( 

.5789 

t 

.IBIS 

) 

.080 

.7109 

( 

.5903 

t 

.8115 

) 

.090 

.7231 

( 

.5994 

i 

.8244 

) 

.100 

.7341 

( 

.6069 

/ 

.8363 

) 

.110 

.7442 

( 

.6132 

1 

.8473 

) 

.120 

.7534 

( 

.6185 

t 

.8574 

) 

.130 

.7619 

( 

.6232 

§ 

.8668 

) 

.140 

.7699 

( 

.6272 

t 

.8754 

) 

.150 

.7773 

( 

.6308 

t 

.8834 

) 

.200 

.8086 

( 

.6444 

/ 

.9154 

) 

.250 

.8331 

( 

.6538 

i 

.9379 

) 

.300 

.8534 

( 

.6609 

t 

.9542 

) 

.400 

.8856 

( 

.6719 

/ 

.9751 

) 

.500 

.9108 

( 

.6807 

/ 

.9868 

) 

.600 

.9316 

( 

.6886 

/ 

.9935 

) 

.700 

.9495 

( 

.6964 

/ 

.9972 

) 

.800 

.9655 

( 

.7048 

/ 

.9990 

) 

.900 

.9806 

( 

.7157 

/ 

.9998 

) 

.950 

.9884 

( 

.7241 

/ 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS; 


EXPECTED  OPERATING  POINT 


( 

FPF  , 

TPF  ) 

( 

( 

.0294, 

.6119) 

( 

( 

.0425, 

.6471) 

( 

( 

.0475, 

.6582) 

( 

( 

,0748, 

.7040) 

( 

LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

.0140, 

.5440) 

(  .0571, 

.6765) 

.0233, 

.5901) 

(  .0726, 

.7010) 

.0271, 

.6040) 

(  .0788, 

.7094) 

.0474, 

.6580) 

(  .1129, 

.7469) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  Input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/20/1998 
TIME  -  12:31:10 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  09  Analog  FAD  resp  for  330  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  308. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =22. 

RESPONSE  DATA: 

CATEGORY  1  2  3  4  5 

ACTUALLY  NEGATIVE  CASES  259.  21.  10.  7.  11. 

ACTUALLY  POSITIVE  CASES  14 .  2 .  2 .  3 .  1 . 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .036  .058  .091  .159  1.000 

TPF:  .000  .045  .182  .273  .364  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.3794  B=  1.5897 

Z(K)=  .998  1.335  1.568  1.803 

LOGL=  -225.2043 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.0584  B=  1.3765 

Z{K)=  1.000  1.314  1.544  1.829 

LOGL=  -224.8260 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.4371 

.2990 

.0142 

.0045 

- .0035 

-  .0136 

B 

.2990 

.2486 

.0040 

-.0046 

- .0116 

-.0202 

z( 

1) 

.0142 

.0040 

.0074 

.0064 

.0057 

.0051 

z( 

2) 

.0045 

-.0046 

.0064 

.0095 

.0088 

.0081 

z( 

3) 

-.0035 

-.0116 

.0057 

.0088 

.0124 

.0116 

z{ 

4) 

- .0136 

- .0202 

.0051 

.0081 

.0116 

.0185 

CORRELATION  MATRIX: 


A 

1.0000 

.9072 

.2495 

.0697 

-.0478 

- .1513 

B 

.9072 

1.0000 

.0940 

-.0953 

-.2092 

-.2982 

z( 

1) 

.2495 

.0940 

1.0000 

.7591 

.5983 

.4319 

z( 

2) 

.0697 

-.0953 

.7591 

1.0000 

.8108 

.6118 

z{ 

3) 

-.0478 

-.2092 

.5983 

.8108 

1.0000 

.7631 

z{ 

4) 

-.1513 

-.2982 

.4319 

.6118 

.7631 

1.0000 
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AREA 


.7330 


STD.  DEV. (AREA) 


.0862 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION; 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.0064 

( 

.0000 

/ 

.1493 

) 

.010 

.0160 

( 

.0004 

t 

.1791 

) 

.020 

.0384 

( 

.0029 

t 

.2192 

) 

.030 

.0629 

( 

.0084 

i 

.2504 

) 

.040 

.0882 

( 

.0172 

f 

.2783 

) 

.050 

.1138 

( 

.0286 

/ 

.3048 

) 

.060 

.1396 

( 

.0421 

# 

.3308 

) 

.070 

.1651 

( 

.0570 

r 

.3569 

) 

.080 

.1905 

( 

.0727 

t 

.3835 

) 

.090 

.2155 

( 

.0887 

t 

.4106 

) 

.100 

.2401 

( 

.1045 

$ 

.4382 

) 

.110 

.2643 

( 

.1198 

9 

.4663 

) 

.120 

.2880 

( 

.1346 

9 

.4946 

) 

.130 

.3113 

( 

.1486 

9 

.5231 

) 

.140 

.3340 

( 

.1618 

9 

.5515 

) 

.150 

.3563 

( 

.1743 

9 

.5795 

) 

.200 

.4602 

( 

.2268 

9 

.7087 

) 

.250 

.5518 

( 

.2674 

9 

.8110 

) 

.300 

.6320 

( 

.3010 

9 

.8841 

) 

.400 

.7612 

( 

.3570 

9 

.9630 

) 

.500 

.8551 

( 

.4062 

9 

.9907 

) 

.600 

.9202 

( 

.4534 

9 

.9983 

) 

.700 

.9624 

( 

.5025 

9 

.9998 

) 

.800 

.9867 

( 

.5580 

9 

1.0000 

) 

.900 

.9976 

( 

.6309 

9 

1.0000 

) 

.950 

.9996 

( 

.6869 

9 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0337, 

.0723) 

(  .0181, 

.0339) 

(  .0592, 

.1375) 

(  .0613, 

.1430) 

(  .0390, 

.0858) 

(  .0924, 

.2216) 

(  .0945, 

.2266) 

(  .0662, 

.1556) 

(  .1307, 

.3130) 

(  .1587, 

.3752) 

(  .1213, 

.2911) 

(  .2029, 

.4657) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  r\m? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader  09  Analog  ROC  Type=diag  N=330  Area=.8645  STD  Area=.0277 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name; 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC 
(y/N,  or  R  to  restart) 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


file  for  the  next  run? 

curve? 

DATE  -  11/20/1998 
TIME  -  12:32:58 


DATA  DESCRIPTION;  Reader  09  Analog  Diag  resp  for  330  cases 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 


NO.  OF  ACTUALLY  NEGATIVE  CASES  =  240. 
NO.  OF  ACTUALLY  POSITIVE  CASES  =  90. 


RESPONSE  DATA: 
CATEGORY 

ACTUALLY  NEGATIVE 
ACTUALLY  POSITIVE 


1  2 

CASES  117.  83. 

CASES  8 .  15 . 


3  4  5 

40.  0.  0. 

41.  13.  13. 


OBSERVED  OPERATING  POINTS: 

FPF:  .000  .000  .000  .167  .512  1.000 

TPF:  .000  .144  .289  .744  .911  1.000 

INITIAL  VALDES  OF  PARAMETERS; 

A=  1.3584  B=  .7683 

Z(K)=  -.031  .967  2.766  2.866 

LOGL=  -387.8173 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 


FINAL  VALDES  OF  PARAMETERS; 

A=  1.2853  B=  .6026 

Z(K)=  -.035  .983  3.076  3.908 

LOGL=  -373.0655 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0317 

.0124 

.0036 

.0032 

-  .0181 

-.0331 

B 

.0124 

.0138 

.0013 

-.0014 

-  .0438 

-.0641 

z( 

1) 

.0036 

.0013 

.0065 

.0035 

-  .0005 

-.0023 

z( 

2) 

.0032 

-.0014 

.0035 

.0090 

.0117 

.0136 

z( 

3) 

-.0181 

-.0438 

-  .0005 

.0117 

.2039 

.2611 

z( 

4) 

-.0331 

-.0641 

-  .0023 

.0136 

.2611 

.3937 

CORRELATION  MATRIX; 

A 

1.0000 

.5914 

.2506 

.1869 

-  .2248 

-.2962 

B 

.5914 

1.0000 

.1360 

-.1216 

-.8269 

-.8700 

z( 

1) 

.2506 

.1360 

1.0000 

.4527 

-  .0129 

- .0447 

z{ 

2) 

.1869 

-.1216 

.4527 

1.0000 

.2724 

.2289 

z( 

3) 

-.2248 

-.8269 

-.0129 

.2724 

1.0000 

.9215 

Z( 

4) 

-  .2962 

-.8700 

-.0447 

.2289 

.9215 

1.0000 
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AREA 


.8645 


STD.  DEV. (AREA) 


0277 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.3947 

( 

.2281  , 

.5835 

) 

.010 

.4535 

( 

.2913  , 

.6241 

) 

.020 

.5189 

( 

.3669  , 

.6682 

) 

.030 

.5603 

( 

.4170  , 

.6960 

) 

.040 

.5910 

( 

.4549  , 

.7168 

) 

.050 

.6156 

( 

.4856  , 

.7337 

) 

.060 

.6362 

( 

.5114  , 

.7479 

) 

.070 

.6539 

{ 

.5336  , 

.7603 

) 

.080 

.6695 

( 

.5531  , 

.7714 

) 

.090 

.6834 

( 

.5705  , 

.7814 

) 

.100 

.6960 

( 

.5862  , 

.7905 

) 

.110 

.7075 

{ 

.6005  , 

.7989 

) 

.120 

.7181 

( 

.6135  , 

.8067 

) 

.130 

.7279 

( 

.6255  , 

.8141 

) 

.140 

.7371 

( 

.6367  , 

.8210 

) 

.150 

,7456 

( 

.6471  , 

.8275 

) 

.200 

.7818 

( 

.6902  , 

.8555 

) 

.250 

.8103 

( 

.7234  , 

.8781 

) 

.300 

.8339 

( 

.7502  , 

.8969 

) 

.400 

.8714 

( 

.7924  , 

.9266 

) 

.500 

.9007 

( 

.8254  , 

.9489 

) 

.600 

.9247 

( 

.8534  , 

.9659 

) 

.700 

.9453 

( 

.8789  , 

.9790 

) 

.800 

.9635 

( 

.9037  , 

.9887 

) 

.900 

,9802 

( 

.9309  , 

.9958 

) 

.950 

.9886 

( 

.9482  , 

.9983 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS; 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

{  .0000, 

.1424) 

(  .0000, 

.0351) 

(  .0037, 

.3713) 

{  .0010, 

.2848) 

(  .0000, 

.1353) 

(  .0142, 

.4860) 

{  .1629, 

.7559) 

(  .1213, 

.7195) 

(  .2128, 

.7897) 

{  .5141, 

.9043) 

(  .4511, 

.8871) 

(  .5768, 

.9195) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader!  1  Analog  ROC  Type 


to-xxw 


NX-  ■X.XX 

XXw'XNxx  "-Vxs.iSJ^lxX^ 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/12/1998 
TIME  -  9:36:45 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  11  Analog  Mass  resp  for  84  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  76. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  8. 

RESPONSE  DATA: 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  62.  8.  1.  2.  3. 

ACTUALLY  POSITIVE  CASES  1.  1.  0.  0.  6. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .039  .066  .079  .184  1.000 

TPF:  .000  .750  .750  .750  .875  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.6573  B=  .5659 

Z(K)=  .899  1.412  1.508  1.757 

LOGL=  -58.4264 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.5802  B=  .5273 

Z(K)=  .898  1.437  1.522  1.736 

LOGL=  -58.3768 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.8191 

.4044 

.0197 

.0029 

-  .0020 

-.0189 

B 

.4044 

.2767 

.0057 

-.0109 

- .0154 

-  .0308 

Z( 

1) 

.0197 

.0057 

.0279 

.0214 

.0206 

.0188 

z{ 

2) 

.0029 

-  .0109 

.0214 

.0446 

.0433 

.0406 

z( 

3) 

-.0020 

-.0154 

.0206 

.0433 

.0493 

.0465 

z( 

4) 

-.0189 

-.0308 

.0188 

.0406 

.0465 

.0661 

CORRELATION  MATRIX: 

A 

1.0000 

.8495 

.1304 

.0152 

-.0099 

-.0813 

B 

.8495 

1.0000 

.0645 

-  .0977 

-.1317 

- .2279 

z{ 

1) 

.1304 

.0645 

1.0000 

.6067 

.5553 

.4370 

z( 

2) 

.0152 

- .0977 

.6067 

1.0000 

.9228 

.7484 

Z( 

3) 

-.0099 

-.1317 

.5553 

.9228 

1.0000 

.8138 

Z{ 

4) 

-.0813 

-.2279 

.4370 

.7484 

.8138 

1.0000 
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AREA  =  .9189  STD .  DEV.  (AREA)  =  .0850 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

/ 

UPPER  BOUND) 

.005 

.5878 

( 

.1038 

/ 

.9558 

) 

.010 

.6381 

( 

.1737 

/ 

.9502 

) 

.020 

.6905 

( 

.2674 

/ 

.9468 

) 

.030 

.7218 

( 

.3289 

t 

.9473 

) 

.040 

.7444 

( 

.3724 

/ 

.9494 

) 

.050 

.7620 

( 

.4043 

f 

.9523 

) 

.060 

.7765 

( 

.4283 

t 

.9556 

) 

.070 

.7887 

( 

.4466 

# 

.9589 

) 

.080 

.7993 

( 

.4608 

/ 

.9622 

) 

.090 

.8087 

( 

.4717 

/ 

.9654 

) 

.100 

.8171 

( 

.4802 

/ 

.9685 

) 

.110 

.8247 

( 

.4867 

/ 

.9713 

) 

.120 

.8316 

( 

.4917 

# 

.9739 

) 

.130 

.8380 

( 

.4954 

/ 

.9764 

) 

.140 

.8439 

( 

.4982 

/ 

.9786 

) 

.150 

.8494 

( 

.5001 

/ 

.9806 

) 

.200 

.8721 

( 

.5012 

/ 

.9884 

) 

.250 

.8897 

( 

.4943 

# 

.9931 

) 

.300 

.9039 

( 

.4834 

/ 

.9960 

) 

.400 

.9260 

( 

.4556 

/ 

.9987 

) 

.500 

.9430 

( 

.4232 

t 

.9996 

) 

.600 

.9567 

( 

.3872 

t 

.9999 

) 

.700 

.9683 

( 

.3466 

/ 

1.0000 

) 

.800 

.9785 

( 

.2986 

t 

1.0000 

) 

.900 

.9880 

( 

.2348 

t 

1.0000 

) 

.950 

.9928 

( 

.1869 

/ 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0413, 

.7469) 

(  .0125, 

.6551) 

(  .1089, 

.8239) 

(  .0640, 

.7816) 

(  .0251, 

.7082) 

(  .1385, 

.8431) 

(  .0753, 

.7946) 

(  .0321, 

.7271) 

(  .1531, 

.8510) 

(  .1846, 

.8658) 

(  .1103, 

.8249) 

(  .2841, 

.8996) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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uoipu  j  aAHisod  «nJi 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  wcuit  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/12/1998 
TIME  -  9:36:45 


R  0  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OP  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  11  Analog  Calc  resp  for  84  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  69. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  15. 

RESPONSE  DATA: 

CATEGORY  1  2 

ACTUALLY  NEGATIVE  CASES  65.  1. 

ACTUALLY  POSITIVE  CASES  8.  1. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .014  .043  .058  1.000 

TPF:  .000  .400  .400  .467  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .3117  B=  .2605 

Z{K)=  1.572  1.712  2.184 

LOGL=  -34.1360 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .3984  B=  .3099 

Z(K)=  1.570  1.792  2.122 

LOGL=  -33.8073 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


3  4  5 

0.  2.  1. 

0.  0.  6. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.4849 

.2138 

.0283 

-.0003 

-.0627 

B 

.2138 

.1205 

.0065 

-.0109 

-.0492 

Z{ 

1) 

.0283 

.0065 

.  .0587 

.0536 

.0472 

z( 

2) 

-.0003 

-.0109 

.0536 

.0758 

.0729 

z( 

3) 

-.0627 

-.0492 

.0472 

.0729 

.1349 

CORRELATION  MATRIX: 

A 

1.0000 

.8844 

.1679 

- .0015 

-  .2451 

B 

.8844 

1.0000 

.0768 

-.1141 

-  .3856 

z{ 

1) 

.1679 

.0768 

1.0000 

.8044 

.5303 

z{ 

2) 

-.0015 

-.1141 

.8044 

1.0000 

.7207 

z( 

3) 

- .2451 

-.3856 

.5303 

.7207 

1.0000 
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AREA  =  .6482  STD.  DEV.  (AREA)  =  .2347 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.3446 

( 

.1077  , 

.6696 

) 

.010 

.3735 

( 

.1440  , 

.6616 

) 

.020 

.4058 

( 

.1833  , 

.6652 

) 

.030 

.4268 

( 

.2045  , 

.6760 

) 

.040 

.4426 

( 

.2173  , 

.6889 

) 

.050 

.4556 

( 

.2253  , 

.7024 

) 

) 

.060 

.4667 

( 

.2304  , 

.7158 

.070 

.4764 

( 

.2335  , 

.7289 

) 

.080 

.4852 

( 

.2352  , 

.7413 

) 

.090 

.4931 

( 

.2361  , 

.7532 

) 

.100 

.5005 

( 

.2362  , 

.7645 

) 

.110 

.5073 

( 

.2359  , 

.7752 

) 

.120 

.5136 

( 

.2352  , 

,7853 

) 

.130 

.5197 

( 

.2342  , 

.7950 

) 

.140 

.5253 

( 

.2329  , 

.8041 

) 

.150 

.5308 

( 

.2316  , 

.8128 

) 

.200 

.5547 

( 

.2232  , 

.8501 

) 

.250 

.5751 

( 

.2138  , 

.8794 

) 

.300 

.5933 

( 

.2042  , 

.9030 

) 

.400 

.6255 

( 

.1853  , 

.9376 

) 

.500 

.6548 

( 

.1669  , 

.9611 

) 

.600 

.6832 

( 

.1486  , 

..9770 

) 

.700 

.7125 

( 

.1297  , 

.9877 

) 

.800 

.7451 

( 

.1092  , 

.9946 

) 

.900 

.7869 

( 

.0839  , 

.9985 

) 

.950 

.8181 

( 

.0663  , 

.9996 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 

(  FPF  ,  TPF  )  (  FPF  ,  TPF  )  (  FPF  ,  TPF  ) 

(  .0169,  .3978)  (  .0022,  .3148)  {  .0805,  .4856) 

{  .0366,  .4376)  (  .0099,  .3729)  (  .1052,  .5041) 

(  .0583,  .4649)  (  .0205,  .4070)  (  .1368,  .5236) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader!  1  Analog  ROC  Typesdiag  N=84  Area=.7269  STD  Area-.0861 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  r\an? 
(y/N,  or  Q  to  c[uit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/12/1998 
TIME  -  10:10:18 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  11  Analog  Diag  resp  for  84  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
{E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  61. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  23. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .000  .016  .115  .295  1.000 

TPF:  .000  .087  .261  .435  .652  1.000 

INITIAL  VALUES  OP  PARAMETERS; 

A=  .8605  B=  .8311 

Z(K)=  .538  1.202  2.135  2.400 

LOGL=  -90.1587 

CHI-SQUARE  GOODNESS  OF  PIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  7  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .7241  B=  .6632 

Z(K)=  .533  1.243  2.072  3.183 

LOGL=  -87.0819 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

43.  11.  6.  1.  0. 

8.  5.  4.  4.  2. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1289 

.0573 

.0220 

.0112 

-.0156 

-.0911 

B 

.0573 

.0525 

.0064 

-.0077 

-.0419 

-.1284 

z( 

1) 

.0220 

.0064 

.0285 

.0199 

.0131 

.0032 

z( 

2) 

.0112 

-.0077 

.0199 

.0416 

.0392 

.0499 

z{ 

3) 

-.0156 

- .0419 

.0131 

.0392 

.1202 

.1744 

Z( 

4) 

-.0911 

- .1284 

.0032 

.0499 

.1744 

.5951 

CORRELATION  MATRIX; 


A 

1.0000 

.6960 

,3631 

.1532 

-.1251 

-.3288 

B 

.6960 

1.0000 

.1657 

-.1654 

- .5270 

-.7262 

z{ 

1) 

.3631 

.1657 

1.0000 

.5779 

.2244 

.0245 

Z{ 

2) 

.1532 

- .1654 

.5779 

1.0000 

.5549 

.3169 

z( 

3) 

-.1251 

-.5270 

.2244 

.5549 

1.0000 

.6523 

z( 

4) 

-.3288 

-.7262 

.0245 

.3169 

.6523 

1.0000 
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AREA 


.7269 


STD.  DEV. (AREA) 


.0861 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.1624 

( 

.0342  , 

.4414 

) 

.010 

.2064 

( 

.0582  , 

.4728 

) 

.020 

.2616 

( 

.0962  , 

.5108 

) 

.030 

.3003 

( 

.1269  , 

.5375 

) 

.040 

.3309 

( 

.1531  , 

.5591 

) 

.050 

.3568 

( 

.1760  , 

.5778 

) 

.060 

.3793 

( 

.1965  , 

.5946 

) 

.070 

.3994 

( 

.2150  , 

.6100 

) 

.080 

.4176 

( 

.2319  , 

.6243 

) 

.090 

.4344 

( 

.2473  , 

.6377 

) 

.100 

.4499 

( 

.2615  , 

.6505 

) 

.110 

.4643 

( 

.2747  , 

.6626 

) 

.120 

.4780 

( 

.2870  , 

.6742 

) 

.130 

.4908 

( 

.2985  , 

.6853 

) 

.140 

.5030 

( 

.3093  , 

.6960 

) 

.150 

.5146 

( 

.3195  , 

.7063 

) 

.200 

.5659 

( 

.3626  , 

.7529 

) 

.250 

.6091 

( 

.3967  , 

.7927 

) 

.300 

.6467 

( 

.4250  , 

.8270 

) 

.400 

.7110 

( 

.4706  , 

.8822 

) 

.500 

.7655 

( 

.5081  , 

.9233 

) 

.600 

.8138 

( 

.5417  , 

.9534 

) 

.700 

.8580 

( 

.5744  , 

.9747 

) 

.800 

.9001 

( 

.6095  , 

.9889 

) 

.900 

.9423 

( 

.6535  , 

.9971 

) 

.950 

.9653 

( 

.6867  , 

.9992 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 


( 

FPF  , 

TPF  ) 

( 

( 

.0007, 

.0827) 

( 

( 

.0191, 

.2578) 

( 

( 

.1069, 

.4600) 

( 

( 

.2970, 

.6445) 

( 

LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

0000, 

,0084) 

(  .0473, 

.3503) 

0030, 

.1355) 

(  .0819, 

.4210) 

0502, 

.3574) 

(  .1996, 

.5655) 

1938, 

.5600) 

(  .4199, 

.7224) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader!  1  digital  ROC  Type=diag  Ns84  Area=.6116  STD  Area=.0907 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/13/1998 
TIME  -  7:40:10 


R  0  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  11  Digital  Diag  resp  for  84  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  61. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  23. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .016  .098  .230  .459  1.000 

TPF:  .000  .174  .348  .435  .609  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .3193  B=  .5844 

Z(K)=  .103  .740  1.291  2.135 

LOGL=  -108.2841 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  3  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .3294  B=  .5907 

Z{K)=  .101  .759  1.266  2.147 

LOGL=  -108.2288 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

33.  14.  8.  5.  1. 

9.  4.  2.  4.  4. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0828 

.0280 

.0154 

.0113 

.0078 

-  .0039 

B 

.0280 

.0357 

.0054 

-.0014 

-  .0089 

-  .0348 

z( 

1) 

.0154 

.0054 

.0257 

.0172 

.0131 

.0074 

z( 

2) 

.0113 

-.0014 

.0172 

.0297 

.0245 

.0212 

Z( 

3) 

.0078 

-.0089 

.0131 

.0245 

.0432 

.0410 

z( 

4) 

-.0039 

-.0348 

.0074 

.0212 

.0410 

.1362 

CORRELATION  MATRIX: 


A 

1.0000 

.5148 

.3329 

.2288 

.1304 

-  .0368 

B 

.5148 

1.0000 

.1791 

-.0445 

-  .2276 

-  .4991 

z( 

1) 

.3329 

.1791 

1.0000 

.6210 

.3928 

.1243 

z( 

2) 

.2288 

- .0445 

.6210 

1.0000 

.6829 

.3337 

z( 

3) 

.1304 

- .2276 

.3928 

.6829 

1.0000 

.5350 

z{ 

4) 

-.0368 

-.4991 

.1243 

.3337 

.5350 

1.0000 
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AREA  =  .6116  STD.  DEV. (AREA)  =  .0907 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION; 


FPF 

TPF 

(LOWER  BOUND, 

UPPER  BOUND) 

.005 

.1166 

( 

.0220  , 

.3553 

) 

.010 

.1480 

( 

.0364  , 

.3835 

) 

.020 

.1884 

( 

.0595  , 

.4171 

) 

.030 

.2172 

( 

.0788  , 

.4403 

) 

.040 

.2404 

( 

.0957  , 

.4589 

) 

.050 

.2603 

( 

.1109  , 

.4748 

) 

.060 

.2779 

( 

.1249  , 

.4890 

) 

.070 

.2937 

( 

.1379  , 

.5019 

) 

.080 

.3083 

( 

.1500  , 

.5139 

) 

.090 

.3218 

( 

.1615  , 

.5251 

) 

.100 

.3344 

( 

.1723  , 

.5358 

) 

.110 

.3464 

( 

.1825  , 

.5460 

) 

.120 

.3577 

( 

.1923  , 

.5557 

) 

.130 

.3684 

( 

.2016  , 

.5651 

) 

.140 

.3788 

( 

.2105  , 

.5742 

) 

.150 

.3887 

( 

.2191  , 

.5831 

) 

.200 

.4334 

( 

.2573  , 

.6242 

) 

.250 

.4726 

( 

.2897  , 

.6615 

) 

.300 

.5079 

( 

.3180  , 

.6960 

) 

.400 

.5714 

( 

.3661  , 

.7587 

) 

.500 

.6291 

( 

.4072  , 

.8142 

) 

.600 

.6840 

( 

.4448  , 

.8635 

) 

.700 

.7386 

( 

.4818  , 

.9072 

) 

.800 

.7957 

( 

.5216  , 

.9451 

) 

.900 

.8614 

( 

.5719  , 

.9768 

) 

.950 

.9034 

( 

.6101  , 

.9899 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .0159, 

.1739) 

(  .0020, 

.0859) 

(  .0772, 

.3044) 

(  .1027, 

.3378) 

(  .0471, 

.2549) 

(  .1952, 

.4294) 

(  .2239, 

.4526) 

(  .1363, 

.3750) 

(  .3367, 

.5321) 

(  .4599, 

.6064) 

(  .3391, 

.5336) 

(  .5846, 

.6757) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  riui? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Type  Ground  Truth,  Mass  Ground  Truth,  uCa  Ground  Truth,  FAD  Ground  Truth,  Dia' 


-  £  o 


tnw  coco  coco 

0  CD  0  0  0  0  0  0 

a:  cc:  o:  a:  a:  o: 
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CM 

CO 
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II 

CO 
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184 

32 

Q. 

CO 

0 

01 

Q.  Q.  Q.  Q.  Q.  Q. 
C/)  W  W  C/)  CO  C/) 

0  0)  0  0  0  0 

QC  01  (x:  a:  cr: 


Appendix  III  -  Reader  14 
Page  2 


Reader14  analog  ROC  Type=mass  N=216  Area=.8737  STD  Area=.0337 


uoipejj  0A!ijsod  enJi 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  wcint  to  save  an  output  file  for  plotting  the  ROC 
(Y/N,  or  R  to  restart) 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 

MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  14  Analog  Mass  resp  for  216  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  184. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  32. 

RESPONSE  DATA: 

CATEGORY  1  2 

ACTUALLY  NEGATIVE  CASES  123.  21. 

ACTUALLY  POSITIVE  CASES  4 .  1 . 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .125  .190  .217  .332  1.000 

TPF:  .000  .656  .813  .844  .875  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.6910  B=  1.0220 

Z(K)=  .435  .781  .877  1.150 

LOGL=  -228.4589 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.7118  B=  1.1132 

Z(K)=  .438  .758  .855  1.162 

LOGL=  -228.1727 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


3  4  5 

5.  12.  23. 

1.  5.  21. 


file  for  the  next  run? 

curve? 

DATE  -  11/  9/1998 
TIME  -  11:49:58 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1918 

.1468 

.0121 

.0091 

B 

.1468 

.1620 

.0047 

-.0006 

z( 

1) 

.0121 

.0047 

.0092 

.0076 

z( 

2) 

.0091 

-.0006 

.0076 

.0102 

Z( 

3) 

.0076 

-  .0030 

.0071 

.0098 

z( 

4) 

,0006 

-.0129 

.0060 

.0086 

CORRELATION  MATRIX: 

A 

1.0000 

.8327 

.2894 

.2050 

B 

.8327 

1.0000 

.1211 

- .0152 

z( 

1) 

.2894 

.1211 

1.0000 

.7803 

z( 

2) 

.2050 

-.0152 

.7803 

1.0000 

Z( 

3) 

.1659 

-.0706 

.7181 

.9279 

z( 

4) 

.0113 

-.2701 

.5321 

.7189 

0076 

.0006 

0030 

-  .0129 

0071 

.0060 

0098 

.0086 

0108 

.0097 

0097 

.0141 

1659 

.0113 

0706 

-.2701 

7181 

.5321 

9279 

.7189 

0000 

.7842 

7842 

1.0000 
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AREA  =  .8737  STD,  DEV.  (AREA)  =  .0337 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

,  UPPER  BOUND) 

.005 

.1238 

( 

.0053 

,  .5966  ) 

.010 

.1898 

( 

.0180 

,  .6327  ) 

.020 

.2826 

( 

.0550 

,  .6729  ) 

.030 

.3511 

( 

.0991 

,  .6991  ) 

.040 

.4061 

( 

.1454 

,  .7194  ) 

.050 

.4524 

( 

.1916 

,  .7365  ) 

.060 

.4923 

( 

.2364 

,  .7515  ) 

.070 

.5273 

( 

.2790 

,  .7651  ) 

.080 

.5586 

( 

.3192 

,  .7778  ) 

.090 

.5867 

( 

.3567 

,  ,7896  ) 

-100 

.6121 

{ 

.3916 

,  .8010  ) 

.110 

.6354 

( 

.4238 

,  .8118  ) 

.120 

.6568 

( 

.4535 

,  .8223  ) 

.130 

.6764 

{ 

.4807 

,  ,8325  ) 

.140 

.6947 

( 

.5057 

,  .8423  ) 

.150 

.7116 

( 

.5286 

,  .8518  ) 

.200 

.7808 

( 

.6168 

,  .8949  ) 

.250 

.8318 

( 

.6743 

,  .9293  ) 

.300 

.8704 

( 

.7142 

,  .9546  ) 

.400 

.9237 

( 

.7671 

,  .9835  ) 

.500 

.9565 

( 

.8033 

,  .9949  ) 

.600 

.9769 

( 

.8322 

,  .9988  ) 

.700 

.9891 

( 

.8579 

,  .9998  ) 

.800 

.9960 

( 

.8830 

,  1.0000  ) 

.900 

.9992 

( 

.9112 

,  1.0000  ) 

.950 

.9998 

( 

.9299 

,  1.0000  ) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

{  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

{  .1226, 

.6621) 

(  .0816, 

.5633) 

{  .1764, 

.7509) 

(  .1961, 

.7762) 

(  .1447, 

.7028) 

(  .2573, 

.8380) 

(  .2243, 

.8074) 

{  .1695, 

.7413) 

(  .2880, 

.8619) 

(  .3306, 

.8895) 

(  .2657, 

.8450) 

(  .4010, 

.9240) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Appendix  HI  -  Reader  14 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/  9/1998 
TIME  -  11:49:58 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  14  Digital  Mass  resp  for  217  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OP  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  184. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  32. 

RESPONSE  DATA; 

CATEGORY  12345 

ACTUALLY  NEGATIVE  CASES  107.  21.  14.  10.  32. 

ACTUALLY  POSITIVE  CASES  2.  0.  5.  1.  24. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .174  .228  .304  .418  1.000 

TPF:  .000  .750  .781  .938  .938  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  2.0380  B=  1.4720 

Z(K)=  .205  .512  .744  .939 

LOGL=  -253.7449 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  5  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.8781  B=  1.2808 

Z(K)=  .207  .491  .767  .939 

LOGL=  -253.2547 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1988 

.1759 

.0120 

.0108 

.0075 

.0039 

B 

.1759 

.2367 

.0053 

.0015 

-.0054 

-.0118 

z( 

1) 

.0120 

.0053 

.0087 

.0072 

.0061 

.0055 

z( 

2) 

.0108 

.0015 

.0072 

.0091 

.0079 

.0072 

z( 

3) 

,0075 

-.0054 

.0061 

.0079 

.0104 

.0097 

z( 

4) 

.0039 

-.0118 

.0055 

.0072 

.0097 

.0118 

CORRELATION  MATRIX: 


A 

1.0000 

.8111 

.2888 

.2534 

.1650 

.0810 

B 

.8111 

1.0000 

.1176 

.0325 

-  .1097 

-.2240 

z{ 

1) 

.2888 

.1176 

1.0000 

.8043 

.6389 

.5411 

z( 

2) 

.2534 

.0325 

.8043 

1.0000 

.8064 

.6959 

Z( 

3) 

.1650 

-.1097 

.6389 

.8064 

1.0000 

.8787 

z{ 

4) 

.0810 

- .2240 

.5411 

.6959 

.8787 

1.0000 
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AREA  =  .8761  STD.  DEV. (AREA)  =  .0344 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.0776 

( 

.0006 

t 

.6553 

) 

.010 

.1352 

( 

.0035 

t 

.6887 

) 

.020 

.2257 

( 

.0177 

f 

.7250 

) 

.030 

.2976 

( 

.0417 

/ 

.7480 

) 

.040 

.3577 

( 

.0731 

f 

.7654 

) 

.050 

.4094 

( 

.1094 

f 

.7798 

) 

.060 

.4547 

( 

.1488 

f 

.7922 

) 

.070 

.4950 

( 

.1899 

t 

.8033 

) 

.080 

.5311 

( 

.2314 

t 

.8134 

) 

.090 

.5638 

( 

.2726 

f 

.8228 

) 

,100 

.5934 

( 

.3127 

r 

.8317 

) 

.110 

.6206 

( 

.3515 

t 

.8402 

) 

.120 

.6454 

( 

.3884 

1 

.8484 

) 

.130 

.6683 

( 

.4235 

/ 

,8562 

) 

.140 

.6895 

( 

.4564 

f 

.8639 

) 

.150 

.7090 

( 

.4872 

/ 

.8714 

) 

.200 

.7882 

( 

.6105 

/ 

.9066 

) 

.250 

.8448 

( 

.6912 

1 

.9370 

) 

.300 

.8863 

( 

.7437 

t 

.9607 

) 

.400 

.9399 

( 

.8055 

t 

.9877 

) 

.500 

.9698 

{ 

.8424 

r 

.9970 

) 

.600 

.9862 

( 

.8693 

t 

.9995 

) 

.700 

.9946 

( 

.8920 

t 

.9999 

) 

.800 

.9984 

( 

.9132 

t 

1.0000 

) 

.900 

.9998 

( 

.9360 

9 

1.0000 

) 

.950 

1.0000 

( 

.9506 

9 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .1739, 

.7503) 

(  .1247, 

.6564) 

(  .2339, 

.8285) 

(  .2215, 

.8147) 

(  .1668, 

.7389) 

(  .2851, 

.8751) 

(  .3115, 

.8941) 

(  .2486, 

.8434) 

(  .3806, 

.9317) 

(  .4180, 

.9466) 

(  .3485, 

.9161) 

(  .4902, 

.9676) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader14analog  ROC  Type=calc  N=216  Area=:.9232  STD  Area-.0229 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/  9/1998 
TIME  -  11:49:58 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  14  Analog  Calc  resp  for  216  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  173. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  43. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .087  .092  .098  .110  1.000 

TPF:  .000  .698  .721  .721  .767  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  2.7379  B=  1.6198 

Z(K)=  1.228  1.292  1.326  1.362 

LOGL=  -110.0205 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  2.6777  B=  1.5878 

Z(K)=  1.227  1.303  1.322  1.361 

LOGL=  -109.7538 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

154.  2.  1.  1.  15. 

10.  2.  0.  1.  30. 


VARIANCE -COVARIANCE  MATRIX: 


A 

2.3862 

1.7890 

.0397 

.0050 

-.0044 

-.0243 

B 

1.7890 

1.3875 

.0116 

-.0160 

-.0235 

-.0392 

z( 

1) 

.0397 

.0116 

.0160 

.0155 

.0153 

.0150 

Z( 

2) 

.0050 

- .0160 

.0155 

.0168 

.0168 

.0168 

z( 

3) 

-.0044 

-.0235 

.0153 

.0168 

.0172 

.0173 

z( 

4) 

-.0243 

-.0392 

.0150 

.0168 

.0173 

.0183 

CORRELATION  MATRIX: 


A 

1.0000 

.9832 

.2030 

.0250 

-.0217 

-.1161 

B 

.9832 

1.0000 

.0779 

-.1046 

-  .1518 

-.2460 

z( 

1) 

.2030 

.0779 

1.0000 

.9416 

.9222 

.8778 

z( 

2) 

.0250 

- .1046 

.9416 

1.0000 

.9882 

.9587 

z( 

3) 

-.0217 

-.1518 

.9222 

.9882 

1.0000 

.9746 

z( 

4) 

- .1161 

-.2460 

.8778 

.9587 

.9746 

1.0000 

Appendix  HI  -  Reader  14 
Page  10 


AREA  =  .9232  STD.  DEV.  (AREA)  =  .0229 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

/ 

UPPER  BOUND) 

.005 

.0789 

( 

.0000 

# 

,9462 

) 

.010 

.1546 

( 

.0003 

f 

.9252 

) 

.020 

.2796 

( 

.0075 

/ 

.8973 

) 

.030 

.3786 

( 

.0373 

t 

.8779 

) 

.040 

.4591 

( 

.0963 

t 

.8639 

) 

.050 

.5261 

( 

.1776 

t 

.8544 

) 

.060 

.5826 

( 

.2686 

t 

.8494 

) 

.070 

.6308 

( 

.3569 

1 

.8496 

) 

.080 

.6723 

( 

.4335 

t 

.8555 

) 

.090 

.7083 

( 

.4934 

! 

.8673 

) 

.100 

.7398 

( 

.5358 

t 

.8840 

) 

.110 

.7673 

( 

.5631 

t 

.9034 

) 

.120 

.7916 

( 

.5791 

i 

.9228 

) 

.130 

.8130 

( 

.5874 

/ 

.9403 

) 

.140 

.8321 

( 

.5905 

f 

.9550 

) 

.150 

.8490 

( 

.5902 

f 

.9668 

) 

.200 

.9101 

( 

.5672 

# 

,9940 

) 

.250 

.9460 

( 

.5332 

/ 

.9991 

) 

.300 

.9675 

( 

.4977 

f 

.9999 

) 

.400 

.9886 

( 

.4287 

# 

1.0000 

) 

.500 

.9963 

( 

.3632 

/ 

1.0000 

) 

.600 

.9990 

( 

.3001 

# 

1.0000 

) 

.700 

.9998 

( 

.2379 

/ 

1.0000 

) 

.800 

1.0000 

( 

.1745 

f 

1.0000 

) 

.900 

1.0000 

( 

.1059 

t 

1.0000 

) 

.950 

1.0000 

( 

.0658 

# 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT  LOWER  BOUND  UPPER  BOUND 


(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

( 

FPF  , 

TPF  ) 

(  .0868, 

.6975) 

(  .0520, 

.5382) 

( 

.1367, 

.8260) 

(  .0931, 

.7187) 

(  .0572, 

.5677) 

( 

.1436, 

.8383) 

(  .0964, 

.7289) 

(  .0597, 

.5815) 

( 

.1472, 

.8445) 

(  .1098, 

.7670) 

(  .0700, 

.6312) 

( 

.1637, 

.8692) 

want  to 

use  data  from 

a  previously 

created 

input 

file  for 

the  next  run? 

(y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 


Appendix  m  -  Reader  14 
Page  11 


ReadeM 4  digital  ROC  Type=calc  N=216  Area=.7734  STD  Area=.6638 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  Wcint  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/  9/1998 
TIME  -  11:49:58 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  14  Calc  resp  for  217  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  173. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  43. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .127  .145  .231  1.000 

TPF:  .000  .535  .558  .651  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .9332  B=  .7422 

Z{K)=  .735  1.060  1.140 

LOGL=  -172.6328 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  3  ITERATIONS . 

FINAL  VALUES  OF  PARAMETERS: 

A=  .9344  B=  .7430 

Z{K)=  .735  1.060  1.140 

LOGL=  -172.6328 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

133.  15.  0.  3.  22. 

15.  4.  0.  1.  23. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.1500 

.1179 

.0112 

.0015 

-.0013 

B 

.1179 

.1258 

.0040 

-  .0074 

- .0108 

ZC 

1) 

.0112 

.0040 

.0111 

.0093 

.0089 

z( 

2) 

.0015 

-.0074 

.0093 

.0136 

.0133 

z{ 

3) 

-.0013 

-.0108 

.0089 

.0133 

.0148 

CORRELATION  MATRIX: 

A 

1.0000 

.8580 

.2737 

.0340 

-  .0279 

B 

.8580 

1.0000 

.1068 

-.1792 

-  .2497 

z{ 

1) 

.2737 

.1068 

1.0000 

.7576 

.6962 

z( 

2) 

.0340 

-.1792 

.7576 

1.0000 

.9434 

z( 

3) 

- .0279 

-.2497 

.6962 

.9434 

1.0000 
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AREA  =  .7734  STD.  DEV.  (AREA)  =  .0638 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE- POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.1636 

( 

.0145  , 

.5890 

) 

.010 

.2135 

( 

.0331  , 

.5979 

) 

.020 

.2770 

( 

.0720  , 

.6093 

) 

.030 

.3216 

( 

.1099  , 

.6181 

) 

.040 

.3570 

( 

.1459  , 

.6258 

) 

.050 

.3867 

( 

.1798  , 

.6332 

) 

.060 

.4126 

( 

.2115  , 

.6403 

) 

.070 

.4355 

( 

.2410  , 

.6475 

) 

.080 

.4563 

( 

.2684  , 

.6548 

) 

.090 

.4753 

( 

.2937  , 

.6623 

) 

.100 

.4929 

( 

.3172  , 

.6700 

) 

.110 

.5092 

( 

.3388  , 

.6779 

) 

.120 

.5245 

( 

.3586  , 

.6861 

) 

.130 

.5388 

( 

.3767  , 

.6946 

) 

.140 

.5524 

( 

.3933  , 

.7034 

) 

.150 

.5653 

( 

.4084  , 

.7124 

) 

,200 

.6214 

( 

.4657  , 

.7595 

) 

.250 

.6677 

( 

.5012  , 

.8062 

) 

.300 

.7072 

( 

.5243  , 

.8483 

) 

.400 

.7723 

( 

.5524  , 

.9133 

) 

.500 

.8249 

( 

.5696  , 

.9548 

) 

.600 

.8691 

( 

.5823  , 

.9792 

) 

.700 

.9072 

( 

.5932  , 

.9921 

) 

.800 

.9406 

( 

.6038  , 

.9979 

) 

.900 

.9704 

( 

.6163  , 

.9997 

) 

.950 

.9845 

( 

.6254  , 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .1272, 

.5349) 

(  .0841, 

.4644) 

(  .1836, 

.6043) 

(  .1445, 

.5583) 

(  .0988, 

.4908) 

(  .2026, 

.6240) 

(  .2312, 

.6512) 

(  .1733, 

.5930) 

(  .2986, 

.7060) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name; 

Do  you  want  to  save  an  output  file  for  plotting 
(Y/N,  or  R  to  restart) 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 

MAXIMUM  LIKELIHOOD  EST 

OF  A  BINORMAL  ROC  C 

FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  14  Analog  FAD  resp  for  216  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  200. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  16. 

RESPONSE  DATA: 

CATEGORY  1  2 

ACTUALLY  NEGATIVE  CASES  150.  9. 

ACTUALLY  POSITIVE  CASES  5.  1. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .145  .195  .205  .250  1. 

TPF:  .000  .563  .563  .625  .688  1. 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.0325  B=  .8519 

Z(K)=  .674  .824  .859  1.058 

LOGL=  -184.7371 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  3  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.0531  B=  .8567 

Z(K)=  .674  .827  .876  1.055 

LOGL=  -184.5440 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


A 

VARIANCE 

.3794 

-COVARIANCE  MATRIX: 
.3188  .0106  .0068 

.0054 

-.0003 

B 

.3188 

.3602 

.0040 

-.0009 

-.0026 

-.0099 

z( 

1) 

.0106 

.0040 

.0093 

.0085 

.0083 

.0075 

z( 

2) 

.0068 

-.0009 

.0085 

.0100 

.0098 

.0090 

z{ 

3) 

,0054 

-.0026 

.0083 

.0098 

.0103 

.0095 

z{ 

4) 

-.0003 

- .0099 

.0075 

.0090 

.0095 

.0119 

A 

CORRELATION  MATRIX: 
1.0000  .8625  .1789 

.1104 

.0869 

-.0040 

B 

.8625 

1.0000 

.0686 

-.0151 

- .0434 

-  .1511 

z( 

1) 

.1789 

.0686 

1.0000 

.8819 

.8452 

.7148 

Z( 

2) 

.1104 

- .0151 

.8819 

1.0000 

.9608 

.8222 

z( 

3) 

.0869 

-.0434 

.8452 

.9608 

1.0000 

.8586 

z{ 

4) 

-.0040 

-.1511 

.7148 

.8222 

.8586 

1.0000 

3  4  5 

2.  10.  29. 

1.  0.  9. 


000 

000 


the  ROC  curve? 

DATE  -  11/  9/1998 
TIME  -  11:49:58 


I  M  A  T  I  O  N 
U  R  V  E 
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AREA  =  .7881  STD.  DEV. (AREA)  =  .0838 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF  TPF  (LOWER  BOUND,  UPPER  BOUND) 


.005 

.1243 

( 

.0006 

# 

.8230 

) 

.010 

.1735 

( 

.0030 

f 

.8057 

) 

.020 

.2399 

( 

.0135 

0 

.7876 

) 

.030 

.2882 

( 

.0301 

0 

.7772 

) 

.040 

.3274 

( 

.0510 

0 

.7706 

) 

.050 

.3608 

( 

.0750 

0 

.7662 

) 

.060 

.3901 

( 

.1009 

0 

.7636 

) 

.070 

.4163 

( 

.1278 

0 

.7624 

) 

.080 

.4400 

( 

.1549 

0 

.7623 

) 

.090 

.4619 

( 

.1818 

0 

.7634 

) 

.100 

.4821 

( 

.2079 

0 

.7654 

) 

.110 

.5009 

( 

.2330 

0 

.7683 

) 

.120 

.5185 

( 

.2568 

0 

.7722 

) 

.130 

.5351 

( 

.2790 

0 

.7769 

) 

.140 

.5507 

( 

.2997 

0 

.7824 

) 

.150 

.5656 

( 

.3187 

0 

.7886 

) 

.200 

.6301 

( 

.3894 

0 

.8278 

) 

.250 

.6828 

( 

.4270 

0 

.8718 

) 

.300 

.7271 

( 

.4443 

0 

.9112 

) 

.400 

.7985 

( 

.4498 

0 

.9640 

) 

.500 

.8538 

( 

.4387 

0 

.9881 

) 

.600 

.8979 

( 

.4200 

0 

.9969 

) 

.700 

.9334 

( 

.3956 

0 

.9995 

) 

.800 

.9620 

( 

.3643 

0 

1.0000 

) 

.900 

.9843 

( 

.3191 

0 

1.0000 

) 

.950 

.9931 

( 

.2823 

0 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .1456, 

.5592) 

(  .1022, 

.4864) 

(  .2000, 

.6301) 

(  .1905, 

.6189) 

(  .1411, 

.5525) 

(  .2492, 

.6819) 

(  .2041, 

.6348) 

(  .1531, 

.5700) 

(  .2640, 

.6958) 

(  .2502, 

.6829) 

(  .1941, 

.6232) 

(  .3138, 

.7381) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  C[uit) 

Stop  -  Program  terminated. 
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Reader14  digital  ROC  Type=fad  N=216  Area=.7236  STD  Area=.0677 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  naune: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 

(Y/N,  or  R  to  restart) 

DATE  -  11/  9/1998 
TIME  -  11:49:58 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  14  Digital  FAD  resp  for  217  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  200. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  16. 

RESPONSE  DATA: 

CATEGORY 

ACTUALLY  NEGATIVE  CASES 
ACTUALLY  POSITIVE  CASES 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .120  .155  .255  .320  1.000 

TPF:  .000  .250  .438  .563  .625  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  .9775  B=  1.2865 

Z(K)=  .467  .659  1.015  1.175 

LOGL=  -233.0557 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  .9963  B=  1.3481 

Z(K)=  .470  .653  .992  1.186 

LOGL=  -232.8176 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


1  2  3  4  5 

136.  13.  20.  7.  24. 

6.  1.  2.  3.  4. 


VARIANCE -COVARIANCE  MATRIX: 


A 

.2528 

.2067 

.0139 

.0113 

.0050 

.0007 

B 

.2067 

.2633 

.0054 

.0013 

-  .0081 

-  .0142 

Z{ 

1) 

.0139 

.0054 

.0085 

.0076 

.0063 

.0057 

z( 

2) 

.0113 

.0013 

.0076 

.0090 

.0076 

.0069 

z( 

3) 

.0050 

-.0081 

.0063 

.0076 

.0111 

.0104 

z( 

4) 

.0007 

-.0142 

.0057 

.0069 

.0104 

.0133 

CORRELATION  MATRIX: 


A 

1.0000 

.8014 

.3002 

.2376 

.0949 

.0121 

B 

.8014 

1.0000 

.1133 

.0266 

-  .1491 

-  .2397 

z( 

1) 

.3002 

.1133 

1.0000 

.8694 

.6472 

.5338 

Z{ 

2) 

.2376 

-0266 

.8694 

1.0000 

.7580 

.6354 

z{ 

3) 

.0949 

-.1491 

.6472 

.7580 

1.0000 

.8542 

z( 

4) 

.0121 

- .2397 

.5338 

.6354 

.8542 

1.0000 
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AREA  = 


.7236 


STD.  DEV. (AREA) 


.0677 


ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

,  UPPER  BOUND) 

.005 

.0066 

( 

.0000 

,  .2804  ) 

.010 

.0162 

( 

.0001 

,  .3150  ) 

.020 

.0381 

( 

.0007 

,  .3567  ) 

.030 

.0618 

( 

.0026 

,  .3860  ) 

.040 

.0862 

( 

.0062 

,  .4098  ) 

.050 

.1109 

( 

.0117 

,  .4306  ) 

.060 

.1356 

(. 

.0191 

,  .4495  ) 

.070 

.1602 

( 

.0284 

,  .4671  ) 

.080 

.1845 

( 

.0395 

,  .4840  ) 

.090 

.2086 

( 

.0522 

,  .5003  ) 

.100 

.2322 

( 

.0663 

,  .5163  ) 

.110 

.2555 

( 

.0815 

,  .5321  ) 

.120 

.2783 

( 

.0975 

,  .5478  ) 

.130 

.3007 

( 

.1142 

,  .5635  ) 

.140 

.3227 

( 

.1313 

,  .5792  ) 

.150 

.3442 

( 

.1487 

,  .5950  ) 

.200 

.4451 

( 

.2326 

,  .6751  ) 

.250 

.5348 

( 

.3041 

,  .7541  ) 

.300 

.6141 

( 

.3611 

,  .8252  ) 

.400 

.7439 

( 

.4439 

,  .9267  ) 

.500 

.8405 

( 

.5043 

,  .9762  ) 

.600 

.9094 

( 

.5550 

,  .9944  ) 

.700 

.9557 

( 

.6026 

,  .9992  ) 

.800 

.9834 

( 

.6524 

,  .9999  ) 

.900 

.9968 

( 

.7134 

,  1.0000  ) 

.950 

.9993 

( 

.7580 

,  1.0000  ) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

(  .1179, 

.2735) 

(  .0790, 

.1823) 

(  .1685, 

.3829) 

(  .1606, 

.3666) 

(  .1154, 

.2679) 

(  .2161, 

.4750) 

(  .2570, 

.5464) 

(  .2008, 

.4466) 

(  .3204, 

.6433) 

(  .3193, 

.6418) 

(  .2577, 

.5475) 

(  .3863, 

.7280) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  rim? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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False  Positive  Fraction 


Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 


(Y/N,  or  R  to  restart) 


DATE  -  11/  9/1998 


TIME  -  11:49:58 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAXIMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  14  Analog  Diag  resp  for  216  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  172. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  44. 

RESPONSE  DATA: 

CATEGORY  123 

ACTUALLY  NEGATIVE  CASES  102.  32.  27. 

ACTUALLY  POSITIVE  CASES  2 .  4 .  13 . 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .017  .064  .221  .407  1.000 

TPF:  .000  .250  .568  .864  .955  1.000 

INITIAL  VALDES  OF  PARAMETERS: 

A=  2.0292  B=  1.2578 

Z(K)=  .235  .769  1.523  2.110 

LOGL=  -256.8944 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALDES  OF  PARAMETERS: 

A=  2.0466  B=  1.2580 

Z(K)=  .237  .763  1.509  2.150 

LOGL=  -256.7898 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


4  5 

8.  3  . 

14.  11. 


VARIANCE -COVARIANCE  MATRIX: 


.1291 

.0699 

.0116 

.0126 

.0044 

-.0163 

.0699 

.0595 

.0037 

.0017 

-.0111 

-  .0330 

1) 

.0116 

.0037 

.0093 

.0068 

.0047 

.0031 

2) 

.0126 

.0017 

,0068 

.0108 

.0081 

.0069 

3) 

.0044 

-.0111 

.0047 

.0081 

.0194 

.0212 

4) 

-.0163 

-.0330 

.0031 

.0069 

.0212 

.0466 

CORRELATION  MATRIX: 

1.0000 

,7972 

.3361 

.3390 

.0887 

-  .2105 

.7972 

1.0000 

.1569 

.0653 

-.3254 

- .6258 

1) 

.3361 

.1569 

1.0000 

.6747 

.3473 

.1482 

2) 

.3390 

.0653 

.6747 

1.0000 

.5570 

.3089 

3) 

.0887 

-.3254 

.3473 

.5570 

1.0000 

.7056 

4) 

-.2105 

-.6258 

.1482 

.3089 

.7056 

1.0000 
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AREA  =  .8986  STD.  DEV. (AREA)  =  .0244 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE-POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE -POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

1 

UPPER  BOUND) 

.005 

.1162 

( 

.0234 

/ 

.3445 

) 

.010 

.1893 

( 

.0574 

t 

.4270 

) 

.020 

.2954 

( 

.1280 

t 

.5243 

) 

.030 

.3745 

( 

.1940 

t 

.5884 

) 

.040 

.4379 

( 

.2536 

t 

.6371 

) 

.050 

.4908 

( 

.3070 

t 

.6766 

) 

.060 

.5360 

( 

.3548 

/ 

.7098 

) 

.070 

.5752 

( 

.3977 

/ 

.7385 

) 

.080 

.6098 

( 

.4362 

/ 

.7636 

) 

.090 

.6405 

( 

.4709 

/ 

.7859 

) 

.100 

.6679 

( 

.5024 

/ 

.8058 

) 

.110 

.6927 

( 

.5309 

/ 

.8236 

) 

.120 

.7151 

( 

.5569 

/ 

.8398 

) 

.130 

.7355 

( 

.5807 

/ 

.8543 

) 

.140 

.7541 

( 

.6026 

/ 

.8676 

) 

.150 

.7712 

( 

.6227 

t 

.8796 

) 

.200 

.8384 

{ 

.7034 

/ 

.9253 

) 

.250 

.8846 

{ 

.7614 

/ 

.9541 

) 

.300 

.9173 

( 

.8053 

/ 

.9722 

) 

.400 

.9580 

( 

.8679 

/ 

.9904 

) 

.500 

.9797 

( 

.9103 

f 

.9970 

) 

-600 

.9910 

{ 

.9407 

1 

.9992 

) 

.700 

.9966 

( 

.9632 

t 

.9999 

) 

.800 

.9990 

{ 

.9799 

i 

1.0000 

) 

.900 

.9999 

{ 

.9920 

t 

i.oooo 

) 

.950 

1.0000 

( 

.9966 

t 

1.0000 

) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 

LOWER 

BOUND 

UPPER 

BOUND 

(  FPF  , 

TPF  ) 

{  FPF  , 

TPF  ) 

{  FPF  , 

TPF  ) 

(  .0158, 

.2551) 

(  .0050, 

.1168) 

{  .0421, 

.4499) 

(  .0657, 

.5591) 

(  .0374, 

.4227) 

(  .1083, 

.6887) 

(  .2226, 

.8613) 

(  .1668, 

.7968) 

(  .2878, 

.9102) 

(  .4062, 

.9598) 

(  .3349, 

.9345) 

(  .4807, 

.9765) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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Reader14  digital  ROC  Type=diag  N=216  Area=.8138  STD  Area=  0372 
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Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(Y/n,  or  Q  to  quit) 
enter  INPUT  file  name: 

Do  you  want  to  save  an  output  file  for  plotting  the  ROC  curve? 


(Y/N,  or  R  to  restart) 


DATE  -  11/  9/1998 


TIME  -  11:49:59 


R  O  C  F  I  T  (IBM  VERSION  1.2)  : 


MAX 


IMUM  LIKELIHOOD  ESTIMATION 
OF  A  BINORMAL  ROC  CURVE 
FROM  RATING  DATA 


DATA  DESCRIPTION:  Reader  14  Digital  Diagnosis  resp  for  217  cases  available 
DATA  COLLECTED  IN  5  CATEGORIES 

WITH  CATEGORY  5  REPRESENTING  STRONGEST  EVIDENCE  OF  POSITIVITY 
(E.G.,  THAT  ABNORMALITY  IS  PRESENT). 

NO.  OF  ACTUALLY  NEGATIVE  CASES  =  172. 

NO.  OF  ACTUALLY  POSITIVE  CASES  =  44. 

RESPONSE  DATA: 

CATEGORY  123 

ACTUALLY  NEGATIVE  CASES  85.  32.  41. 

ACTUALLY  POSITIVE  CASES  4.  5.  18. 

OBSERVED  OPERATING  POINTS: 

FPF:  .000  .023  .081  .320  .506  1.000 

TPF:  .000  .273  .386  .795  .909  1.000 

INITIAL  VALUES  OF  PARAMETERS: 

A=  1.2816  B=  1.0046 

Z(K)=  -.015  .468  1.396  1.991 

LOGL=  -280.7432 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 

PROCEDURE  CONVERGES  AFTER  4  ITERATIONS. 

FINAL  VALUES  OF  PARAMETERS: 

A=  1.2701  B=  1.0134 

Z(K)=  -.018  .467  1.446  1.898 

LOGL=  -280.0007 

CHI-SQUARE  GOODNESS  OF  FIT  NOT  CALCULATED 
BECAUSE  SOME  EXPECTED  CELL  FREQUENCIES  ARE  LESS  THAN  5. 


4  5 

10.  4. 

5.  12 . 


VARIANCE -COVARIANCE  MATRIX: 


A 

.0651 

.0311 

.0085 

.0084 

.0042 

-.0013 

B 

.0311 

.0337 

.0031 

.0014 

-.0084 

-  .0168 

z( 

1) 

.0085 

.0031 

.0091 

.0065 

.0038 

.0028 

Z( 

2) 

.0084 

.0014 

,0065 

.0095 

.0061 

.0054 

z( 

3) 

.0042 

-.0084 

.0038 

,0061 

.0187 

.0188 

z( 

.4) 

-.0013 

-.0168 

.0028 

.0054 

.0188 

.0330 

CORRELATION  MATRIX: 

A 

1.0000 

.6646 

.3512 

.3388 

.1191 

-.0281 

B 

.6646 

1.0000 

.1785 

.0765 

-  .3355 

-.5049 

z( 

1) 

.3512 

.1785 

1.0000 

.7065 

.2935 

-1621 

z( 

2) 

.3388 

.0765 

.7065 

1.0000 

.4620 

.3031 

Z( 

3) 

.1191 

-.3355 

.2935 

.4620 

1.0000 

.7566 

z  ( 

4) 

-.0281 

-.5049 

.1621 

.3031 

.7566 

1.0000 
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AREA  =  .8138  STD.  DEV.  (AREA)  =  .0372 

ESTIMATED  BINORMAL  ROC  CURVE,  WITH  LOWER  AND  UPPER 
BOUNDS  ON  ASYMMETRIC  95%  CONFIDENCE  INTERVAL  FOR 
TRUE -POSITIVE  FRACTION  AT  EACH  SPECIFIED 
FALSE-POSITIVE  FRACTION: 


FPF 

TPF 

(LOWER  BOUND 

,  UPPER  BOUND) 

.005 

.0900 

( 

.0206 

,  .2615  ) 

.010 

.1384 

( 

.0431 

,  .3227  ) 

.020 

.2085 

( 

.0864 

,  .3976  ) 

.030 

.2623 

( 

.1263 

,  .4489  ) 

.040 

.3070 

{ 

.1631 

,  .4893  ) 

.050 

.3457 

( 

.1972 

,  .5230  ) 

.060 

.3799 

{ 

.2288 

,  .5522  ) 

.070 

.4107 

( 

.2583 

,  .5782  ) 

.080 

.4388 

( 

.2858 

,  .6017  ) 

.090 

.4646 

( 

.3116 

,  .6231  ) 

.100 

.4885 

( 

.3358 

,  .6430  ) 

.110 

.5108 

( 

.3587 

,  .6614  ) 

.120 

.5316 

( 

.3802 

,  .6786  ) 

.130 

.5512 

( 

.4005 

,  .6947  ) 

.140 

.5696 

( 

.4198 

,  .7099  ) 

.150 

.5870 

( 

.4380 

,  .7243  ) 

.200 

.6618 

( 

.5170 

,  .7859  ) 

.250 

.7214 

( 

.5803 

,  .8343  ) 

.300 

.7701 

( 

.6326 

,  .8728  ) 

.400 

.8447 

( 

.7153 

,  .9277  ) 

.500 

.8980 

( 

.7794 

,  .9616  ) 

.600 

.9365 

{ 

.8319 

,  .9817  ) 

.700 

,9642 

( 

.8770 

,  .9927  ) 

.800 

.9831 

( 

.9171 

,  .9979  ) 

.900 

.9949 

( 

.9547 

,  .9997  ) 

.950 

.9983 

{ 

.9739 

,  1.0000  ) 

ESTIMATES  OF  EXPECTED  OPERATING  POINTS  ON  FITTED  ROC 
CURVE,  WITH  LOWER  AND  UPPER  BOUNDS  OF  ASYMMETRIC  95% 
CONFIDENCE  INTERVAL  ALONG  THE  CURVE  FOR  THOSE  POINTS: 


EXPECTED  OPERATING  POINT 


{ 

FPF  , 

TPF  ) 

( 

( 

.0288, 

.2568) 

( 

{ 

.0741, 

.4227) 

( 

( 

.3203, 

.7873) 

{ 

( 

.5074, 

.9013) 

( 

LOWER 

BOUND 

UPPER 

BOUND 

FPF  , 

TPF  ) 

(  FPF  , 

TPF  ) 

.0121, 

.1553) 

(  .0615, 

.3849) 

.0433, 

.3204) 

(  .1194, 

.5305) 

.2555, 

.7271) 

{  .3912, 

.8390) 

.4331, 

.8642) 

(  .5814, 

.9303) 

Do  you  want  to  use  data  from  a  previously  created  input  file  for  the  next  run? 
(y/N,  or  Q  to  quit) 

Stop  -  Program  terminated. 
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